Eye and / or pupil tracking method, eye and / or pupil tracking device and virtual retinal display

Abstract

The invention relates to an eye and / or pupil tracking method, in particular for data glasses (10), wherein an infrared laser beam (12) is scanned over an eye (20) with the aid of a microscanner (18) comprising a micromirror (14, 16), and wherein images (24, 26) are created from infrared reflection signals (22) of the infrared laser beam (12) reflected by the eye (20), on the basis of which the eye and / or pupil positions of the eye (20) can be determined. It is proposed that at least during flybacks (28) of a scan cycle (30) of the microscanner (18), in which at least one of the micromirrors (14) of the microscanner (18) is returned to a scan start position (32), the infrared laser beam (12) and an infrared detection element (34) for detecting the infrared reflection signals (22) are activated, preferably remain activated, and an image (24) is recorded on the basis of the reflection signals (22) thereby detected, from which the instantaneous eye and / or pupil position can be determined.

Description

State of the art

[0001] An eye and / or pupil tracking method has already been proposed, wherein an infrared laser beam is scanned over an eye using a microscanner comprising micromirrors, and wherein images are created from infrared reflection signals of the infrared laser beam reflected by the eye, on the basis of which eye and / or pupil positions of the eye can be determined. Disclosure of the invention

[0002] The invention is based on an eye and / or pupil tracking method, in particular for data glasses, wherein an infrared laser beam is scanned over an eye with the aid of a microscanner comprising micromirrors, and wherein images are created from infrared reflection signals of the infrared laser beam reflected by the eye, on the basis of which eye and / or pupil positions of the eye can be determined.

[0003] It is proposed that at least during flybacks of a scan cycle of the microscanner, in which at least one of the micromirrors of the microscanner is returned to a scan start position, in particular the slow mirror of the microscanner is returned to the scan start position of the slow mirror, the infrared laser beam and an infrared detection element for detecting the infrared reflection signals are activated, preferably remain activated, and based on the reflection signals detected thereby, an image is recorded from which the current eye and / or pupil position can be determined. This can advantageously increase the temporal resolution of pupil tracking, in particular in data glasses with a virtual retinal display comprising an integrated infrared eye tracking system. A mandatory flyback of a microscanner can advantageously be used to obtain information.Advantageously, a prediction of a pupil position for a subsequent regular scan of the microscanner is enabled. This advantageously allows for faster and / or more accurate regulation and / or control of the microscanner, particularly in data glasses with a virtual retinal display that uses a moving beamlet approach for eyebox maximization. The scan start position corresponds in particular to the position assumed by the micromirrors of the microscanner at the beginning of each scan cycle.

[0004] In particular, the eye and / or pupil tracking method is intended for determining and / or tracking a position and / or a gaze vector of an eye of a user of data glasses. The term “data glasses” is to be understood in particular as a wearable (head-mounted display) by means of which information can be added to a user’s field of vision. Data glasses preferably enable augmented reality, virtual reality and / or mixed reality applications. Data glasses are also commonly referred to as smart glasses or VR glasses or AR glasses. In particular, the data glasses have the virtual retinal display (also called retinal scan display or light field display), which is particularly familiar to those skilled in the art. The virtual retinal display is in particular designed to sequentially display an image content by deflecting at least one visible laser beam from at least one temporally modulated light source, such asone or more (RGB) laser diodes of a laser projector, scanned and projected by optical elements directly onto the retina of the user's eye. The eye and / or pupil tracking method can be used in particular for eye tracking by an eye tracking system that is preferably integrated into the data glasses. An eye tracking system is particularly configured to track and / or record a movement of at least the eye and / or at least the pupil of the user. "Configured" or "intended" should be understood in particular to mean specially programmed, designed and / or equipped. The fact that an object is intended or configured for a specific function should be understood in particular to mean that the object fulfills and / or executes this specific function in at least one application and / or operating state.

[0005] The eye and / or pupil tracking method, in particular the eye tracking system, could be provided for detecting an eye and / or pupil position of the user's eye by means of the so-called "dark pupil effect". Preferably, however, the eye and / or pupil tracking method, in particular the eye tracking system, is provided for detecting an eye and / or pupil position of the user's eye by means of the so-called "bright pupil effect", which is based in particular on the relatively high infrared reflectivity of the retina of the eye compared to neighboring / other components of the user's eye.

[0006] When using the "bright pupil effect," the pupil of the eye can be determined by eye and / or pupil tracking directly from an image captured by an infrared detection element, which is embodied as a photodiode integrated into an infrared laser module that also emits the infrared laser beam, e.g., as a laser feedback interferometry (LFI) sensor. In particular, the retina generates a so-called laser speckle pattern upon reflection of a laser beam, especially a sufficiently coherent laser beam. The laser speckle pattern (also called laser granulation or light granulation) is, in particular, a grainy-looking interference phenomenon that can be observed with sufficiently coherent illumination of optically rough object surfaces, such as the retina of the eye (cf. cones and rods of the retina).The LFI sensor for utilizing the "bright pupil effect" could be designed as a VCSEL, preferably a ViP-VCSEL ("vertical-cavity surface-emitting laser with integrated photo diode"). The LFI sensor could be integrated into the data glasses, e.g., into a frame, lens, or temple of the data glasses. The LFI sensor could be integrated into a common laser module / laser projector with the data glasses' (RGB) laser diodes that generate the visible image content of the virtual retinal display. Alternatively, the LFI sensor could also be designed separately from the laser projector with the (RGB) laser diodes that generate the visible image content. In this case, the infrared laser beam of the LFI sensor could be coupled into the RGB laser beam of the (RGB) laser diodes that generate the visible image content via a coupling structure of the data glasses, in particular one designed separately from the (RGB) laser projector.It is also conceivable for the data glasses to have more than one LFI sensor, e.g., two, three, or more than three LFI sensors. The exploitation of the "bright pupil effect" is known to those skilled in the art; see, for example, the publication by J. Meyer et al., "A Highly Integrated Ambient Light Robust Eye-Tracking Sensor for Retinal Projection AR Glasses Based on Laser Feedback Interferometry," from ETRA '22. In particular, the same infrared detection element that is provided for determining the eye and / or pupil position is also provided for the continued infrared measurement during the flyback. The photodiode for detecting the infrared reflection signal is preferably sampled at equidistant intervals with a fixed sampling rate.

[0007] When using the "dark pupil effect," the light from the infrared laser beam scattered back by the eye, particularly the infrared reflection signal, is detected by an infrared detection element designed as an external, off-axis photodiode, which is designed and arranged separately from the laser module that emitted the infrared laser beam. The pupil signal appears dark in the external photodiode because the pupil acts like an aperture for the infrared laser beam scanned across the eye, and the infrared laser beam penetrates the pupil, meaning only a small amount of scattered light from the infrared laser beam in the area of ​​the pupil can reach the off-axis external photodiode. The off-axis scattering of the infrared laser beam is significantly greater in the parts of the eye surrounding the pupil, where the infrared light cannot penetrate the eye, than in the area of ​​the pupil.The use of the "dark pupil effect" is known to those skilled in the art (see, for example, the publication by J. Meyer, T. Schlebusch, W. Fuhl, and E. Kasneci, "A Novel Camera-Free Eye Tracking Sensor for Augmented Reality Based on Laser Scanning," in IEEE Sensors Journal, Vol. 20, No. 24, pp. 15204-15212, 2020). In both cases—when using the "dark pupil effect" and when using the "bright pupil effect," image processing algorithms, such as a Canny Edge detection algorithm, can be used to detect the pupil contour in the respective images, and a pupil ellipse can then be determined. The eye and / or pupil positions of the eye can then be determined from this data.

[0008] In particular, the same microscanner, preferably the same micromirror that scans the infrared laser beam across the eye, also projects the visible images of the virtual retinal display in the data glasses by scanning a visible (RGB) laser beam, preferably writing them onto the retina of the user's eye. The microscanner represents, in particular, a micro-opto-electro-mechanical system from the class of micromirror actuators for dynamic light modulation. The microscanner is a MEMS microscanner. The microscanner forms a MEMS micromirror system, in particular a 2x1D MEMS micromirror system with two micromirrors. The two micromirrors are preferably pivotable in different (mutually perpendicular) directions.A modulating movement of individual mirrors of the microscanner can occur in a rotational manner around at least one or two axes, which in particular deflects light beams incident on the individual mirrors. In particular, the microscanner has two separately movable individual mirrors. One of the two individual mirrors is a fast mirror, which is preferably actuated via a sawtooth pattern. One of the two individual mirrors is a fast mirror, which is preferably operated resonantly with a sine pattern. In particular, the scanned laser beam is scanned line by line, preferably sinusoidally. However, column by column, particularly sinusoidally, scanning of the scanned laser beam is also conceivable. Scanning frequencies of the microscanner are in the range of several kilohertz, in particular in the range up to 50 kHz. The flyback of the scan cycle of the microscanner is significantly faster than the regular forward scan (cf.Actuation with sawtooth pattern). During the flyback, only the slow mirror of the microscanner is reset. However, it is also conceivable that both micromirrors are reset during the flyback. However, the fast mirror of the microscanner preferably continues to oscillate continuously in a resonant manner during the flyback. In particular, the slow mirror periodically alternates between forward scans and flybacks. A scan cycle of the microscanner preferably comprises exactly one forward scan and exactly one flyback of the slow mirror. In particular, the slow mirror usually scans vertically / perpendicular to the scan lines generated by the microscanner. In particular, the slow mirror ensures a change of the scan line during image formation by the microscanner. In particular, the fast mirror scans horizontally / along the scan lines generated by the microscanner. In particular, the fast mirror generates the scan lines during image formation.A scan cycle of the microscanner preferably comprises a plurality of oscillations of the fast mirror. A scan cycle of the microscanner preferably comprises a number of oscillations of the fast mirror corresponding to half the number of image lines of an image generated by the virtual retinal display.

[0009] To eliminate possible deformations in a pupil contour acquired during the fast flyback, an image correction algorithm is used to calculate these deformations of the acquired pupil. From the pupil contours thus determined, the pupil position can be extracted, for example, by determining / fitting a pupil center within the acquired and preferably corrected pupil contour. The scan cycle is repeated periodically. Based on the mirror positions of the micromirrors, a two-dimensional infrared reflectivity map is preferably generated, from which the pupil contours can then be determined using the image processing algorithms.

[0010] It is further proposed that, at least during forward scans of the scanning cycle of the microscanner, in particular of the slow mirror of the microscanner, in which visible images are projected by means of the microscanner, the infrared laser beam and an infrared detection element for detecting the infrared reflection signals are activated, in particular in addition to a visible laser beam for image generation. Based on the infrared reflection signals thus detected, a further image is recorded from which the current eye and / or pupil position can be determined. This advantageously allows a particularly high temporal resolution of pupil tracking to be achieved. The forward scan, in particular, forms the regular scan of the laser system of the virtual retinal display for image generation.In particular, during the forward scan of the microscanner's scan cycle, in particular of the microscanner's slow mirror, and during the flyback of the microscanner's scan cycle, in particular of the microscanner's slow mirror, a separate image of the pupil is recorded from the infrared reflection signals reflected by the eye. From each of these images, a current eye and / or pupil position can be determined, preferably independently of one another. In particular, two images of the eye / pupil are thus created per scan cycle of the microscanner / per frame of the virtual retinal display.

[0011] Furthermore, it is proposed that the image have a lower resolution than the other image. This advantageously allows for a particularly high temporal resolution of pupil tracking. Advantageously, the proposed invention allows two separate images of the pupil to be captured within a complete frame of the virtual retinal display, and accordingly, a movement of the pupil within this single frame can be detected.

[0012] Furthermore, it is proposed that a laser beam intended to generate a visible image be deactivated during the flyback. This advantageously allows for high energy efficiency. In particular, the visual impression for the user is not impaired, especially since visible light on the retina does not produce any visual impression during the short period of the fast flyback (backscan) of the slow mirror, and correct image information cannot be displayed during this time.

[0013] Furthermore, it is proposed that at least the current eye and / or pupil position determined for one of the flybacks be used as input for predicting a future eye and / or pupil position during a forward scan of the microscanner, in particular of the slow mirror of the microscanner, following the flyback, in particular directly. This advantageously allows for optimizing, in particular increasing, the temporal resolution and / or accuracy of the eye and / or pupil tracking. The following forward scan is part of a next scan cycle of the microscanner / a next frame of the virtual retinal display.

[0014] In addition, it is proposed that an eye and / or pupil velocity of the eye be estimated, in particular determined, based on a synopsis of the two eye and / or pupil positions determined for a forward scan of the microscanner, in particular of the slow mirror of the microscanner, and the associated flyback. This advantageously allows a temporal resolution and / or accuracy of the eye and / or pupil tracking to be optimized, in particular increased. In addition, further information for predicting the future eye and / or pupil position can advantageously be obtained. Advantageously, a ballistic eye model can be used to estimate the point at which the eye is currently located in a typical course of a movement saccade.The eye and / or pupil velocity is preferably determined from a difference signal formation of the two scans of the scan cycle, i.e. the forward scan and the flyback of the scan cycle.

[0015] Furthermore, it is proposed that a known velocity profile of a ballistic eye movement model be used to predict the eye and / or pupil position. This advantageously allows a particularly high prediction accuracy to be achieved. The ballistic velocity profile is preferably stored on a storage unit of the data glasses or on an external storage unit to which the data glasses have access, e.g., a cloud or an external mobile device. Velocity profiles of ballistic eye movement models are known; see, for example, the publications by Frederich K. Hsu, A. Terry Bahill, and Lawrence Stark, "Parametric sensitivity analysis of a homeomorphic model for saccadic and vergence eye movements," in Computer Programs in Biomedicine 6.2, pp. 108-116, 1976, or by H. Collewijn, C.J. Erkelens, and R.M. Steinman."Binocular coordination of human horizontal saccadic eye movements," in The Journal of Physiology 404.1, pp. 157-182, 1988. The velocity curve measured using the eye and / or pupil tracking method is preferably compared with the known ballistic velocity profile. Based on the curve progression, the further course of the eye and / or pupil movement trajectory can then be predicted.

[0016] It is also proposed that the position of an eyebox of a virtual retinal display, in particular of data glasses, be changed based on the predicted eye and / or pupil positions. This advantageously makes it possible to achieve a particularly user-friendly virtual retinal display with a particularly fast response time when the viewing angle changes. The position of the eyebox could, for example, be changed using a controllable tilt mirror (“moving beamlet principle”). Starting from an initial position of the eye / a last measured eye position, a new optimal tilt mirror position, in which, for example, the pupil is in the center of the eyebox, is determined using the eye and / or pupil tracking method, and the eyebox is then moved to this new tilt mirror position until the eyebox matches the new, changed eye position.If the eye now moves further or again, the updated pupil position and pupil velocity are measured and fed back as a deviation from the target value. In addition to the measured pupil position, the measured pupil velocity is also taken into account, so that the deviation already corresponds to the target position, taking into account the latencies required to deflect the mirror and the upstream conversion step of the eye. A visible image, specifically an RGB image, is projected from the virtual retinal display into the eyebox. Visible images projected into the eyebox are perceptible to the wearer of the data glasses.An “eyebox” of a virtual retinal display and / or data glasses is understood to mean, in particular, a spatial area within which all light rays of the scanning projection of the visible laser beams of the virtual retinal display can pass through an entrance pupil of the eye of the user of the virtual retinal display and / or data glasses.

[0017] Furthermore, an eye and / or pupil tracking device and a virtual retinal display and / or data glasses with an eye and / or pupil tracking device are proposed, preferably for carrying out the eye and / or pupil tracking method, wherein the eye and / or pupil tracking device has at least one laser projector unit, which is at least configured to generate an infrared laser beam and to scan an eye with the aid of a micro-scanner of the laser projector unit comprising a micro-mirror, wherein the eye and / or pupil tracking device has at least one detector unit, which is at least configured to detect infrared reflection signals of the infrared laser beam reflected by the eye to create images from which eye and / or pupil positions of the eye can then be determined, and wherein at least during flybacks of a scan cycle of the micro-scanner,in which at least one of the micromirrors of the microscanner is reset to a scan start position, the infrared laser beam of the laser projector unit and an infrared detection element of the detector unit are activated, preferably remain activated, to detect the infrared reflection signals, and an image can be created based on the reflection signals thus detected, from which the current eye and / or pupil position can be determined. This can advantageously increase the temporal resolution of pupil tracking. Advantageously, a mandatory flyback of a microscanner can be used to obtain information. Advantageously, a prediction of a pupil position for a subsequent regular scan of the microscanner is enabled. Advantageously, faster and / or more accurate regulation and / or control of the microscanner can be achieved.

[0018] The eye and / or pupil tracking device, in particular the virtual retinal display and / or the data glasses, has a computing unit for carrying out the eye and / or pupil tracking method. A "computing unit" is to be understood in particular as a unit with an information input, an information processing unit, and an information output. The computing unit advantageously has at least one processor, a memory, input and output means, further electrical components, an operating program, control routines, control routines, and / or calculation routines. The components of the computing unit are preferably arranged on a common circuit board and / or advantageously arranged in a common housing. Alternatively, the computing unit can be embodied as an external computing unit, e.g., a cloud or a mobile device, which is in communication with the virtual retinal display and / or the data glasses.

[0019] The eye and / or pupil tracking method, the eye and / or pupil tracking device, and the virtual retinal display according to the invention are not intended to be limited to the application and embodiment described above. In particular, the eye and / or pupil tracking method, the eye and / or pupil tracking device, and the virtual retinal display according to the invention may, in order to fulfill a functionality described herein, have a number of individual elements, components, units, and method steps that differs from the number stated herein. Furthermore, in the value ranges specified in this disclosure, values ​​within the stated limits are also to be considered disclosed and can be used arbitrarily. drawing

[0020] Further advantages will become apparent from the following description of the drawings. The drawings illustrate an exemplary embodiment of the invention. The drawings, the description, and the claims contain numerous features in combination. Those skilled in the art will also expediently consider the features individually and combine them into useful further combinations.

[0021] They show: Fig. 1 schematically shows a virtual retinal display of data glasses with an eye and / or pupil tracking device, Fig. 2 schematically shows a sequence of a scanning cycle of a microscanner of the virtual retinal display and exemplary images taken by the eye and / or pupil tracking device during the scanning cycle, Fig. 3 a schematic flow diagram of an eye and / or pupil tracking method, Fig. 4 different velocity profiles of a ballistic eye movement model taken from the literature Fig. 5 a speed-distance diagram with an exemplary eye trajectory determined by means of the eye and / or pupil tracking method and Fig. 6 shows an exemplary control loop based on the measurement results of the eye and / or pupil tracking method. Description of the embodiment

[0022] The Fig. 1 schematically shows a virtual retinal display 44. The virtual retinal display 44 forms part of a pair of data glasses 10. Only one lens 52 of the data glasses 10 is shown. The virtual retinal display 44 comprises an eye and / or pupil tracking device 50. The eye and / or pupil tracking device 50 is provided for carrying out an eye and / or pupil tracking method. The eye and / or pupil tracking device 50 comprises a laser projector unit 46. The laser projector unit 46 is configured to generate an infrared laser beam 12. The laser projector unit 46 comprises an infrared laser diode 54. The infrared laser diode 54 is part of a laser feedback interferometry (LFI) sensor 56. The infrared laser diode 54 is integrated into a laser module 58 of the laser projector unit 46.An alternative infrared laser diode 54' and / or an alternative LFI sensor (not shown) could also be formed separately from the laser module 58 (see dashed box with the reference numeral 54' in the . Fig. 1). In this alternative case, the virtual retinal display 44 could have a coupling structure 60 that focuses the infrared laser beam 12 with a visible laser beam 38 of the virtual retinal display 44. The laser projector unit 46 comprises RGB laser diodes 102 for generating and outputting the visible laser beam 38 of the virtual retinal display 44. The visible laser beam 38 is intended to generate an image visible to the user, which is written directly line by line onto the retina of an eye 20 of the user.

[0023] The laser projector unit 46 comprises a microscanner 18. The microscanner 18 is designed as a 2x1D MEMS mirror system. The microscanner 18 comprises micromirrors 14, 16. A micromirror 14 of the microscanner 18 is a slow micromirror that vertically scans images 24, 26 generated by the virtual retinal display 44. A micromirror 16 of the microscanner 18 is a fast micromirror that horizontally scans images 24, 26 generated by the virtual retinal display 44, thus generating the individual image lines of the images 24, 26. The laser projector unit 46 is configured to scan the infrared laser beam 12 across the user's eye 20 using the microscanner 18. Alternatively, a 2D deflection system other than the microscanner 18 could be used to generate the same inventive effect.

[0024] Possible options include a 2D MEMS mirror, a spatial light modulator, a rotating prism or mechanically actuated mirrors.

[0025] The virtual retinal display 44 comprises a (mechanical) tilt mirror 62. The tilt mirror 62 is arranged behind the microscanner 18 in the optical path of the light emitted by the laser module 58. The tilt mirror 62 is provided for displacing an eyebox 42 of the virtual retinal display 44. Based on position data of the pupil of the eye 20 determined by the eye and / or pupil tracking device 50, the tilt mirror 62 is provided to only displace the eyebox 42 such that it always overlaps with the pupil of the eye 20. The tilt mirror 62 forms the central component for implementing the moving beamlet approach. The tilt mirror 62 is provided to deflect a two-dimensional beam field generated by the microscanner 18 in two dimensions.

[0026] The virtual retinal display 44 has an optical system 64, which, for example, comprises two optical elements. Alternative optical systems with more or fewer than two optical elements are also conceivable. Fig. In the case illustrated in Figure 1, the optical system 64 has a projection lens 66. The two-dimensional beam field deflected by the tilt mirror 62 is directed via the projection lens 66 onto the spectacle lens 52 of the data glasses 10. The spectacle lens 52 of the data glasses 10 comprises a further optical element of the optical system 64, which is integrated into the spectacle lens 52 of the data glasses 10. Alternatively, the further optical element could also be arranged separately from the spectacle lens 52. The further optical element of the spectacle lens 52 is designed as a diffractive optical element (DOE) or as a holographic optical element (HOE). The DOE / HOE has the task of redirecting the incident light toward the eye 20. An optical deflection function of the DOE / HOE for the light of the visible laser beam 38 differs from an optical deflection function of the DOE / HOE for the light of the infrared laser beam 12.The light of the visible laser beam 38 is redirected convergently so that the visible laser beams 38 collect in the pupil of the eye 20. The light of the infrared laser beam 12 is redirected less convergently, parallel, or even divergently, so that it illuminates a larger area in an eye plane of the virtual retinal display 44.

[0027] The eye and / or pupil tracking device 50 includes a detector unit 48. The detector unit 48 is configured to detect infrared reflection signals 22 reflected by an eye 20 of a user of the virtual retinal display 44. The detector unit 48 includes an infrared detection element 34. The infrared detection element 34 is configured as an infrared photodiode or a CCD sensor or the like. The infrared detection element 34 is part of the LFI sensor 56. In this case, the infrared detection element 34 measures the infrared reflection signal 22 reflected back by the retina of the eye 20 in the region of the pupil. Alternatively, the infrared detection element 34' could be configured externally to the laser module 58. In this case, the infrared detection element 34' could be integrated into a spectacle frame (not shown) or into the spectacle lens 52.In this case, the infrared detection element 34' predominantly measures the infrared reflection signal 22 scattered by the outer surface of the eye. If the eye 20 moves, the eyebox 42, in particular the cone of the visible laser beam 38 in the eye plane of the virtual retinal display 44, must be shifted so that colored light and thus visible image content continue to fall into the pupil, thus creating a perceptible image on the retina. For this purpose, the point of incidence of the laser beams 12, 38 on the optical element integrated in the spectacle lens 52 can be shifted. For this purpose, the tilt mirror 62 and / or the projection optics with the projection lens 66 are used.The tilt mirror 62 is rotated, for example, so that the two-dimensional scan field of the microscanner 18 is deflected differently, allowing the eyebox 42 / beam cone to follow the pupil and allowing visible light projected from the laser projector unit 46 at a new pupil position to enter the eye 20. Fig. 1 shows two different positions of the tilt mirror 62 with the respective beam paths and eyeboxes 42. In the Fig. 1 also shows two exemplary viewing positions of the eye 20, which correspond to the respective positions of the tilt mirror 62. For the correct tracking of the tilt mirror 62, information about the position of the pupil is required. This is determined from the light path of the infrared laser beam 12 using the eye and / or pupil tracking method according to the invention.

[0028] The Fig. 2 schematically shows a sequence of a scan cycle 30 of the microscanner 18 of the virtual retinal display 44 and exemplary images 24, 26 that are recorded by the infrared detection element 34 of the eye and / or pupil tracking device 50 during the scan cycle 30. In a center of the Fig. 2 shows the exemplary images 24, 26.

[0029] Below images 24, 26, an angle-time diagram 68 of a micromirror 14, in particular the slow mirror, of the microscanner 18 is shown. In the angle-time diagram 68, a horizontal deflection angle α of the micromirror 14 is plotted against time t. The angle-time diagram 68 shows a progression of the angular position of the micromirror 14 within a single scan cycle 30. This single scan cycle 30 is continuously repeated during normal operation of the virtual retinal display 44. The single scan cycle 30 comprises exactly one forward scan 36. During the forward scan 36, the angular deviation of the micromirror 14 increases from its initial position to a value α maxThe individual scan cycle 30 comprises exactly one flyback 28. During the flyback 28, the micromirror 14, in particular the slow mirror, of the microscanner 18 is returned to a scan start position 32. The scan start position 32 corresponds to the angular position α0. During the flyback 28, the angular deviation of the micromirror 14 decreases again until it finally returns to the initial position that the micromirror 14 also held at the beginning of the forward scan 36. The time period for the forward scan 36 is significantly and substantially (many times) longer than the time period for the flyback 28. This creates a sawtooth pattern of the angle-time dependence of the micromirror 16.

[0030] To the right of Figures 24, 26, an angle-time diagram 70 of a further micromirror 16, in particular the fast mirror, of the microscanner 18 is shown. In the angle-time diagram 70, a vertical deflection angle β of the further micromirror 16 is plotted against time t. The angle-time diagram 70 shows a progression of the angular position of the further micromirror 16 within the flyback 28. During the flyback 28, the further micromirror 16 oscillates several times between its maximum deflections β0 and β max . The further micromirror 16, in particular the fast mirror, also performs the same oscillation movement during the forward scan 36 (not shown).

[0031] The number of oscillations of the further micromirror 16, in particular of the fast mirror, is higher than during the flyback 28, since the total time of the forward scan 36 is higher than that of the flyback 28.

[0032] The further micromirror 16, in particular the fast mirror, is operated resonantly. The further micromirror 16, in particular the fast mirror, scans in a sinusoidal pattern along the horizontal axis β between β0 and β max while the micromirror 14, in particular the slow mirror, is simultaneously actuated via the sawtooth pattern. The micromirror 14, in particular the slow mirror, scans vertically linearly with the angle α between α0 and α maxThus, a 2D field is scanned in the pupil plane. In parallel, the infrared detection element 34 is scanned, and based on the current scanning angles of the microscanner 18, the two-dimensional images 24, 26 are created. Image 24 is created during the forward scan 36, and image 26 is created during the flyback 28. In each of these two images 24, 26, pupil detection can be performed using known pupil detection algorithms. A pupil center can be determined from each of these two images 24, 26. Image 24 has a lower resolution than the other image 26.

[0033] The Fig. 3 shows a schematic flow diagram of the eye and / or pupil tracking method for the data glasses 10. In the eye and / or pupil tracking method, the images 24, 26 are created from the infrared reflection signals 22 of the infrared laser beam 12 reflected by the eye 20, based on which images the eye and / or pupil positions of the eye 20 in the data glasses 10 can then be determined at any given time. The virtual retinal display 44 and / or the data glasses 10 have a computing unit 104 (cf. Fig. 1) to carry out the eye and / or pupil tracking method. In at least one method step 72, the infrared laser beam 12 is generated and output by the laser projector unit 46. In method step 72, the visible laser beam 38 is also generated and output by the laser projector unit 46. In at least one method step 74, the infrared laser beam 12 is scanned over the eye 20 with the aid of the microscanner 18 comprising the micromirrors 14, 16 and with the aid of the optical system 64. In method step 74, the visible laser beam 38 is also scanned over the eye 20 with the aid of the microscanner 18 comprising the micromirrors 14, 16 and with the aid of the optical system 64. The infrared laser beam 12 and the visible laser beam 38 are scanned together in method step 74.In method step 74, the laser beams 12, 38 are scanned according to the fixed, repeated scan cycle 30, which includes the forward scan 36 and the flyback 28, in particular consisting of the forward scan 36 and the flyback 28. During the forward scan 36, the visible laser beam 38 and the infrared laser beam 12 are activated. During the flyback 28, the infrared laser beam 12 remains activated. During the flyback 28, the infrared laser beam 12 is output by the laser projector unit 46. During the flyback 28, the visible laser beam 38 is deactivated. During the flyback 28, the visible laser beam 38 is not output by the laser projector unit 46.

[0034] In at least one further method step 76, the infrared reflection signals 22 are detected and recorded by the detector unit 48. In method step 76, infrared reflection signals 22 are recorded during the flyback 28 and during the forward scan 36, and the associated images 24, 26 are created therefrom. In at least one further method step 78, the current eye and / or pupil position is determined from the images 24, 26. In method step 78, a future eye and / or pupil position of a forward scan 36 directly following the flyback 28 is predicted based on the image 24 recorded during a flyback 28. In method step 78, the instantaneous eye and / or pupil position determined for this flyback 28 is used as input for the prediction of the future eye and / or pupil position during the forward scan 36 of the microscanner 18 directly following the flyback 28.For this purpose, a first position C. x1 , C y1 , which was recorded during the forward scan 36, and a second position C x2 , C y2 , which was recorded during the flyback 28, as input for the prediction of the next pupil position C x_t+1 , C y_t+1 For this purpose, in a sub-step 80 of process step 78, the second position C x2 , C y2 by an image correction algorithm so that distortions in the image 24, which may occur due to the rapid backscan in the flyback 28, are corrected. Subsequently, in a further sub-step 82 of method step 78, the thus corrected pupil center C x2' , C y2' together with the position of the forward scan 36 C x1 , C y1transformed into a common image coordinate system in the pupil plane. After the transformation into the image coordinate system of the pupil plane, a pupil velocity v x , v y be determined. In sub-step 84, an eye and / or pupil velocity of the eye 20 can thus be estimated, in particular determined, based on a synopsis of the two eye and / or pupil positions determined for the forward scan 36 and the associated flyback 28. In at least one further sub-step 86 of method step 78, a speed profile 40, in particular a known one (cf. Fig. 4) a known ballistic eye movement model is used. The eye 20 is modeled as a dynamic system and described using known eye model parameters of the known ballistic eye movement models. A model of the dynamics of the eye 20 can be implemented, for example, as described below: Jθ¨=Fag−Fant−Kpθeye−Bpθ˙eye−Bagθ˙ag−Bantθ˙ant Theta eye describes a rotation angle of the eye 20 in the horizontal direction, Θ ag and θ ant the angular velocities of the two responsible muscles of the eye 20. In addition, the two forces F ag and F ant of the respective muscles of the eye. In the literature, various values ​​for the constant variables of the model can be found, specifically the inertia J, the spring constant K and the damping constants B ag and B ant .

[0035] In the Fig. 4 shows various velocity profiles 40 of the ballistic eye movement model taken from the literature. The velocity profiles 40 correspond to different eye rotation angles (the degrees of the eye rotation angles corresponding to the velocity profiles 40 are shown in the Fig. 4).

[0036] In the Fig. 5 shows a velocity-displacement diagram 88 with an exemplary eye trajectory 90 determined using the previously described eye and / or pupil tracking method. The crosses are intended to indicate specific velocity measurement points of the eye and / or pupil velocity of the eye 20, which were obtained using the previously described eye and / or pupil tracking method. From a comparison or fit of the measurement points to the known velocity profiles 40 associated with sub-step 86, the best-fitting velocity profile 40 can be determined from a set of known velocity profiles 40 (cf. Fig. 4) and the eye rotation angle associated with the observed movement is determined. Since the eye 20 moves on the ballistic velocity curve (in mm / s) of the selected velocity profile 40 in order to reach a position (in mm) in the pupil plane, the further course of the movement trajectory of the eye 20 can be predicted based on the previously determined velocity measurement points of the eye and / or pupil velocity of the respective eye movement together with the known position of the pupil at the beginning of the eye movement (in the Fig. 5 shown in dashed lines).

[0037] In at least one further method step 92, a position of the eyebox 42 of the virtual retinal display 44, in particular of the data glasses 10, is changed based on the predicted eye and / or pupil positions. To change the position of the eyebox 42, the tilt mirror 62 can be used, which is controlled by a control loop 94 (see FIG. Fig. 6) based on the results of the previously described eye and / or pupil tracking procedure.

[0038] The Fig.6 shows, by way of example, the control loop 94 with which the tilt mirror 62 is operated to adjust the position of the eyebox 42. Starting from an initial position of the eye 20 / a last determined position of the eye 20, a new position of the tilt mirror 62 is determined in a control step 96 based on the results of the eye and / or pupil tracking method. In a further control step 98, this new position of the tilt mirror 62 is then moved to until the (RGB) eyebox 42 matches the new position of the eye 20. If the eye 20 now moves again, the updated pupil position and the pupil velocity are measured using the eye and / or pupil tracking method in a further control step 100 and fed back as a deviation from the target value.In addition to the measured pupil position, the speed of the pupil is also taken into account, so that the deviation already corresponds to the target position, taking into account the required latencies for the deflection of the mirror and for the position calculation. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] J. Meyer et. al., “A Highly Integrated Ambient Light Robust Eye-Tracking Sensor for Retinal Projection AR Glasses Based on Laser Feedback Interferometry,” from ETRA'22

[0006] von J. Meyer, T. Schlebusch, W. Fuhl and E. Kasneci, „A Novel Camera-Free Eye Tracking Sensor for Augmented Reality Based on Laser Scanning,“ in IEEE Sensors Journal, Vol. 20, No. 24, pp. 15204-15212, 2020

[0007] Frederich K. Hsu, A.Terry Bahill und Lawrence Stark. „Parametric sensitivity analysis of a homeomorphic model for saccadic and vergence eye movements“, in Computer Programs in Biomedicine 6.2, S. 108-116, 1976

[0015] H Collewijn, C J Erkelens und R M Steinman. „Binocular co-ordination of human horizontal saccadic eye movements“, in The Journal of Physiology 404.1, S. 157-182, 1988

[0015]

Claims

[1] Eye and / or pupil tracking method, in particular for data glasses (10), wherein an infrared laser beam (12) is scanned over an eye (20) with the aid of a micro-scanner (18) comprising a micro-mirror (14, 16), and wherein images (24, 26) are created from infrared reflection signals (22) of the infrared laser beam (12) reflected by the eye (20), on the basis of which eye and / or pupil positions of the eye (20) can be determined, characterized bythat at least during flybacks (28) of a scan cycle (30) of the microscanner (18), in which at least one of the micromirrors (14) of the microscanner (18) is returned to a scan start position (32), the infrared laser beam (12) and an infrared detection element (34) for detecting the infrared reflection signals (22) are activated, preferably remain activated, and based on the reflection signals (22) detected thereby, an image (24) is recorded from which the current eye and / or pupil position can be determined. [2] Eye and / or pupil tracking method according to claim 1, characterized bythat at least during forward scans (36) of the scan cycle (30) of the microscanner (18), in which visible images are projected by means of the microscanner (18), the infrared laser beam (12) and an infrared detection element (34) for detecting the infrared reflection signals (22), in particular in addition to a visible laser beam (38) for image generation, are activated, and on the basis of the infrared reflection signals (22) detected thereby, a further image (26) is recorded, from which the current eye and / or pupil position can be determined. [3] Eye and / or pupil tracking method according to claim 1 and 2, characterized by that the image (24) has a lower resolution than the other image (26). [4] Eye and / or pupil tracking method according to one of the preceding claims, characterized by that a visible laser beam (38) intended to generate a visible image is deactivated during the flybacks (28). [5] Eye and / or pupil tracking method according to one of the preceding claims, characterized by that at least the current eye and / or pupil position determined for one of the flybacks (28) is used as input for a prediction of a future eye and / or pupil position during a forward scan (36) of the microscanner (18) following the flyback (28), in particular directly. [6] Eye and / or pupil tracking method according to at least one of claims 2 or 3, characterized by that an eye and / or pupil speed of the eye (20) is estimated, in particular determined, based on a synopsis of the two eye and / or pupil positions determined for a forward scan (36) of the microscanner (18) and the associated flyback (28). [7] Eye and / or pupil tracking method according to claims 5 and 6, characterized bythat a, in particular known, speed profile (40) of a ballistic eye movement model is used to predict the eye and / or pupil position. [8] Eye and / or pupil tracking method according to one of claims 5 or 7, characterized by that a position of an eyebox (42) of a virtual retinal display (44), in particular of the data glasses (10), is changed based on the predicted eye and / or pupil positions. [9] Eye and / or pupil tracking device (50), in particular data glasses (10), preferably for carrying out an eye and / or pupil tracking method according to one of the preceding claims, with at least one laser projector unit (46) which is at least configured to generate an infrared laser beam (12) and to scan an eye (20) with the aid of a micro-scanner (18) of the laser projector unit (46) comprising a micro-mirror (14, 16), and with at least one detector unit (48) which is at least configured to detect infrared reflection signals (22) of the infrared laser beam (12) reflected by the eye (20) in order to create images (24, 26) from which eye and / or pupil positions of the eye (20) can then be determined, characterized bythat at least during flybacks (28) of a scan cycle (30) of the microscanner (18), in which at least one of the micromirrors (14, 16) of the microscanner (18) is returned to a scan start position (32), the infrared laser beam (12) of the laser projector unit (46) and an infrared detection element (34) of the detector unit (48) are activated, preferably remain activated, for detecting the infrared reflection signals (22), and an image (24) can be created on the basis of the reflection signals (22) detected thereby, from which image the current eye and / or pupil position can be determined. [10] Virtual retinal display (44), in particular data glasses (10), with an eye and / or pupil tracking device (50) according to claim 9.

Images