Method for operating a device for tracking the viewing direction of a person, device for tracking the viewing direction of a person, and smart glasses

Abstract

The invention relates to a method for operating a device (130) for tracking the viewing direction of a person. The method has an illuminating step, a reading step, a dividing step, and a changing step. In the illuminating step, an eye region of the person is illuminated with an illumination light. In the reading step, an image signal is read which represents an image of reflections of an eye region of the person. In the dividing step, the image is divided into a plurality of regions (150) on the basis of a reflection behavior in the regions in question. In the changing step, the power of the illumination light is changed in at least one of the regions (150).

Description

[0001] Description

[0002] title

[0003] Method for operating a device for one person to one a person and data glasses

[0004] State of the art

[0005] The invention relates to a method for operating a device for tracking a person's gaze direction, a device for tracking a person's gaze direction, and data glasses according to the preamble of the independent claims. The present invention also relates to a computer program.

[0006] An infrared laser can be used in an LFI (laser feedback interferometry) approach to collect information from the surface the laser can scan. The laser beam can be propagated via two mirrors to the eye area, which can interact with the laser beam. From the angles of the mirrors at a given time, an image can be reconstructed, which can show the intensity of the backscattered light at a specific point on the scanned surface. Static noise may be present in the image.

[0007] Disclosure of the invention

[0008] Against this background, the approach presented here presents a method for operating a device for tracking a person's gaze direction, a device for tracking a person's gaze direction, and data glasses, as well as a corresponding computer program according to the main claims. The measures listed in the dependent claims enable advantageous refinements and improvements of the device specified in the independent claim.

[0009] The advantages achievable with the approach presented here consist in particular in the creation of a method that can advantageously change the performance of the device.

[0010] A method for operating a device for tracking a person's gaze direction is presented. The method comprises an illuminating step, a reading step, a subdividing step, and a modifying step. In the illuminating step, an eye region of the person is illuminated with an illuminating light. In the reading step, an image signal representing an image of reflections of an eye region of the person is read in. In the subdividing step, the image is divided into a plurality of regions depending on a reflection behavior in the respective regions. In the modifying step, a power of the illuminating light is modified in at least one of the regions.

[0011] The illumination light can be a laser beam, more specifically an infrared laser. The approach presented here can prevent unwanted interactions in the image, such as static noise. Furthermore, scratches in the glass and unwanted glitter can be detected and divided into regions accordingly. A region can be divided, for example, into a pupil region or a noise region. The approach presented here can also be understood as feature-dependent optical performance for pupil detection. The reflectance can represent the amount of returned light, with high reflectance representing a high amount of returned light and low reflectance representing a low amount of returned light.

[0012] In the subdividing step, the image can be divided into regions where the reflection behavior can be the same within a tolerance range. In the subdividing step, a region in the image can be interpreted as a disturbance region, in which a disturbance on an optical surface in a light path may be located if the reflection behavior in the disturbance region does not change over a predefined period of time. In the changing step, the power of the illumination light in the disturbance region can be reduced. For example, the reflection behavior in such a disturbance region can include pixels whose reflection behavior has a light reflection that is increased by a reflection threshold compared to other pixels outside the disturbance region.The reflection threshold can, for example, be at least 20 percent higher than the reflection of incident light, so that pixels in the interference region reflect at least 20 percent more incident light than pixels outside the interference region. The interference region may contain scratches, dust, or other defects that can be detected by the reflection behavior. By reducing the power, energy can be saved, allowing the device to operate more efficiently and have a longer lifespan.

[0013] In the subdividing step, a region in the image can be interpreted as a pupil region, in which a pupil of a person's eye can be imaged if the reflection behavior of pixels in the pupil region changes from an initial value and subsequently returns to the initial value. In the changing step, the power of the illumination light in the pupil region can be increased. By increasing the power, the device can be operated advantageously, as relevant information can be reflected into the eye in a way that is more easily recognizable for the user of the device.

[0014] In the subdividing step, a region can be created as a horizontal stripe of the subject's eye area. This allows for a simple and technically inexpensive subdivision of the image.

[0015] In the reading step, an image signal can be read in, which can represent a filter image determined using low-pass filtering from a plurality of images of the person's eye area. In particular, the plurality of images of the person's eye area were taken while the person had their eyes closed.

[0016] The method may include a step of illuminating the person's eye area with a homogeneous light intensity if no pupil of the person has been detected. Subsequently, the steps of reading, subdividing, and modifying may be performed repeatedly. In the repeated reading step, an image signal may be read in that may represent a person's eye area illuminated with the homogeneous light intensity.

[0017] This method can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control unit.

[0018] The approach presented here further provides a device designed to perform, control, or implement the steps of a variant of a method presented here in corresponding devices. This embodiment of the invention in the form of a device also allows the problem underlying the invention to be solved quickly and efficiently.

[0019] For this purpose, the device can have at least one computing unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to a sensor or an actuator for reading sensor signals from the sensor or for outputting data or control signals to the actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The computing unit can be, for example, a signal processor, a microcontroller, or the like, wherein the memory unit can be a flash memory or a magnetic storage unit.The communication interface can be designed to read in or output data wirelessly and / or wired, wherein a communication interface that can read in or output wired data can read this data, for example, electrically or optically from a corresponding data transmission line or output it to a corresponding data transmission line.

[0020] In this case, a device can be understood as an electrical device that processes sensor signals and outputs control and / or data signals depending on them. The device can have an interface, which can be implemented in hardware and / or software. In a hardware implementation, the interfaces can, for example, be part of a so-called system ASIC, which contains a wide variety of functions of the device. However, it is also possible for the interfaces to be separate integrated circuits or to consist at least partially of discrete components. In a software implementation, the interfaces can be software modules that are present, for example, on a microcontroller alongside other software modules.

[0021] Data glasses comprise one embodiment of a device mentioned herein. Data glasses can be a visual output device that can be worn on the nose like conventional glasses. The lenses of the glasses can display screens or display devices for displaying virtual images.

[0022] Also advantageous is a computer program product or computer program with program code that can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out, implement and / or control the steps of the method according to one of the embodiments described above, in particular when the program product or program is executed on a computer or a device.

[0023] Examples of the approach presented here are shown in the drawings and explained in more detail in the following description. It shows:

[0024] Fig. 1 shows a schematic representation of an embodiment of data glasses; Fig. 2 shows a representation of an image for an embodiment of a device;

[0025] Fig. 3 shows a representation of an image for an embodiment of a device;

[0026] Fig. 4 shows a representation of an image for an embodiment of a device;

[0027] Fig. 5 shows a representation of an image for an embodiment of a device;

[0028] Fig. 6 shows a representation of an image for an embodiment of a device;

[0029] Fig. 7 is a flowchart of an embodiment of a method for operating a device for tracking a person's gaze direction; and

[0030] Fig. 8 is a block diagram of an embodiment of a device for operating a device for tracking a line of sight of a person.

[0031] In the following description of advantageous embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and having a similar effect, whereby a repeated description of these elements is omitted.

[0032] Fig. 1 shows a schematic representation of an embodiment of data glasses 100. The data glasses 100 are designed, for example, to be worn by a person on the nose like conventional glasses.

[0033] The data glasses 100 comprise, merely by way of example, two lenses 105, 110, which are connected to each other via a bridge 115. The lenses 120, 125 are designed, for example, as screens or display devices for displaying virtual images. Furthermore, the data glasses 100 comprise two temples 120, 125 for wearing the data glasses 100.

[0034] The data glasses 100 have a device 130 which is arranged on the one temple 120 merely by way of example.

[0035] The device 130 is configured, for example, to track a person's line of sight 144. For this purpose, the device 130 is configured, merely by way of example, as a laser device and / or includes a photodiode. The device 130 is configured, for example, to detect and scan a region 132 of the glass 105.

[0036] The device 130 is designed to scan at least one eye 135 of the person and / or the eye-tracking area 132, for example, using an infrared reader. The photodiode serves, for example, to detect light reflected by the eye 135.

[0037] In an operational state of the device 130, an eye area of ​​the person is illuminated with an illuminating light.

[0038] Next, an image is taken that represents reflections from the person's eye area. The reflections are then differentiated into desired reflections and undesired reflections.

[0039] Desired reflections are, for example, reflections of a pupil 140 of the eye 135, while undesired reflections are, for example, scratches 145 in the glass 105 of the data glasses 100.

[0040] The image is then divided into a plurality of regions, depending on the reflection behavior of the respective regions. In Fig. 1, the image is divided into a region 150 for illustrative purposes only.

[0041] Using the regions, the power of the device 100 and / or the illumination light is changed in at least one of the regions. According to one embodiment, when dividing the image into different regions, a region is interpreted as a disturbance region if there is a disturbance, such as scratches 145 and / or dust, on the glass 105. In this case, the power of the device 130 and / or the illumination light is reduced in the disturbance region.

[0042] According to a further embodiment, when dividing the image into different regions, a region 150 is interpreted as a pupil region if the pupil 140 of the person's eye 135 is located in the region 150. In this case, the power of the device 130 and / or the illumination light is increased in the interference region. In Fig. 1, the region 150 is interpreted as a pupil region merely by way of example.

[0043] The approach presented here thus enables the power of device 130 and / or the illumination light to be changed depending on whether a pupil 140 or a disturbance in region 150 of the image has been detected. A pupil region is of significantly greater interest to device 130 than a disturbance region. By reducing the power of device 130 and / or the illumination light, for example, static noise is reduced, a signal-to-noise ratio is improved, and / or power consumption is reduced.

[0044] In other words, Fig. 1 describes a method for controlling the output power of a device 130 for eye tracking, which scans an eye 135 and its surroundings. This device can also be referred to as an infrared laser and has an internal photodiode for detecting, for example, via interference, light reflected from the eye 135. The method either identifies or excludes regions of the scanned area 132 that, apart from the desired reflection 155 of a pupil 140 of the eye 135, exhibit unwanted reflections caused by scratches 145 in the glass 105 of the data glasses 100, optical elements such as holograms and lenses, etc. The regions are identified, for example, when the eye is closed and does not itself produce any prominent reflections 155 from the pupil 140, "red eyes."Region exclusion can be achieved by, for example, excluding the region around a last known pupil position. In the identified or excluded regions, the laser output power is then reduced to reduce static noise, improve the signal-to-noise ratio, reduce power consumption, etc.

[0045] In other words, IR laser diodes are tuned to a specific power level, 500 pW for example, which is important because the optical performance of the data glasses 100 is well defined. A combination of RGB and infrared, or IR for short, is used here. This upper limit is untouchable.

[0046] But there's always a way to reduce the optical power in regions we're not interested in. These regions can be divided into static and dynamic imperfections.

[0047] Static imperfections are the glitter and other effects that arise from the optical path itself and do not change over time. Dynamic imperfections are effects that originate from the person's eye area, such as piercings, sunscreen, etc. A secondary area in the nose area also contributes, as there is always a secondary reflection from the lens covering the holographic optical element, or HOE for short. These are quasi-static and shift over time, as lens slippage still exists.

[0048] There are two approaches to classifying the regions of interest:

[0049] To find all "bad spots" when no pupil signal is present, we take information from the image when the subject has their eyes closed. This happens regularly. In this case, no pupil is detected, but we still see a signal or several signals in various regions of the image. These include both the imperfections of the optical path and the user characteristics in the entire eye region. A low-pass filter on all images taken over time when the user has their eyes closed allows us to classify static and dynamic imperfections. Another approach is to control the optical power based on the last known position of the pupil 140.This approach establishes a closed control loop. As soon as pupil information is lost, for example due to blinking or a change in speed, the power is increased again to ensure correct detection.

[0050] In this way, the optical power is reduced everywhere except in the pupil area, or, if optical switching of the optical power costs too much, the maximum power is used only in the horizontal pupil plane, see for example Fig. 5. This reduces the total signal everywhere except in the pupil area / plane.

[0051] With the data glasses 100 presented here and / or the device 130, which can also be referred to as an optical power control system, additional dependencies on the secondary side can also be added. Depending on the duty cycle of the device 130, which can also be referred to as an IR laser, the cavity temperature changes. Based on the cavity temperature, the wavelength also shifts slightly, with a higher temperature resulting in a higher wavelength. This can be used to better match the IR laser to the holographic optical element.

[0052] The main features of the approach presented here are:

[0053] Depending on detected features in the image and / or the last known pupil position, the optical power can be reduced if no pupil is expected there. Depending on the desired wavelength, the optical power remains high or is reduced more drastically.

[0054] Depending on the desired temperature, the optical power is reduced, which increases the service life.

[0055] The approach presented here makes it possible to reduce flicker and other unwanted information in the area located at the beginning of the process chain. Static noise is also reduced. Furthermore, the dynamic range of the pupil signal is increased, thereby increasing the signal-to-noise ratio (SNR) of the desired signal. The laser cavity temperature can be reduced, and the wavelength can also be easily adjusted to enable better matching with the holographic optical element. Furthermore, power consumption is reduced.

[0056] The approach presented here thus enables a dynamic adjustment of the optical power based on spatial features, where the pupil 140 is a positive feature and imperfections such as scratches 145 are a negative feature.

[0057] Fig. 2 shows a representation of an image 200 for an embodiment of a device. The device is similar to or corresponds to the device in Fig. 1.

[0058] More specifically, Fig. 2 shows a raw image when the subject closes at least one eye. Some disturbances 205 can be seen in the image 200, for example, scratches and / or dust.

[0059] Fig. 3 shows an illustration of an image 300 for an embodiment of a device. The device is similar to or corresponds to the device of Fig. 1. The image 300 is similar to or corresponds to the image of Fig. 2, except that the interference 205 is masked out. Additionally, a pupil signal 305 is shown, which represents a position of the pupil, more precisely a reflection 155 of the pupil, despite the eye being closed. This is possible, for example, if a last known position of the pupil 140 is used as a reference.

[0060] In other words, Fig. 3 shows the pupil signal 305. The disturbances 205 are masked out in this case. In the poor regions, i.e., the regions where the disturbances 205 were detected, the laser power can be reduced, thereby suppressing the signal strength of the defects. Fig. 4 shows an illustration of an image 400 for an embodiment of a device. The device is similar to or corresponds to the device from one of the figures described above.

[0061] The image 400 depicts the pupil 140 or the reflection of the pupil 140. The image 400 shows the area 132 captured by the device before the image 400 is divided into a plurality of regions.

[0062] Fig. 5 shows a representation of an image 400 for one embodiment of a device. The image 400 is similar or corresponds to the image in Fig. 4, except that the image 400 or area 132 is divided into at least one region 500.

[0063] According to one embodiment, the region 500 is identified by the device as a pupil region 500. The pupil region 500 extends over a horizontal area of ​​the image 400.

[0064] For example, the regions 505, 510 outside the pupil region 500 are identified by the device as interference regions 505, 510, with the pupil region 500 being arranged between the interference regions 505, 510 merely by way of example. The device is designed such that a higher power is output in the pupil region 500 than in the interference regions 505, 510.

[0065] Fig. 6 shows a representation of an image 400 for one embodiment of a device. The image 400 is similar or identical to the image in Fig. 5, except that the pupil region 500 is depicted differently.

[0066] The pupil region 500 is rectangular in shape merely by way of example. According to one embodiment, the pupil region 500 has a tolerance range around the pupil. The pupil region 500 is surrounded merely by way of example by an interference region 505, wherein the device has a higher performance in the pupil region 500 than in the interference region 505. Fig. 7 shows a flowchart of an embodiment of a method 700 for operating a device for tracking a person's gaze direction.

[0067] The method 700 includes a step 705 of illuminating, a step 710 of reading, a step 715 of subdividing, and a step 720 of modifying. Optionally, the method 700 includes a step 725 of illuminating.

[0068] In step 705 of illuminating, an eye area of ​​the person is illuminated with an illuminating light.

[0069] In step 710 of reading, an image signal is read in that represents an image of reflections of an eye region of the person. According to one embodiment, the image signal represents a filter image. The filter image was determined using low-pass filtering from a plurality of images of the eye region of the person. In particular, the plurality of images of the eye region of the person were taken while the person had their eyes closed.

[0070] In subdividing step 715, the image is divided into a plurality of regions depending on the reflectance behavior in the respective regions. According to one embodiment, the image is divided into regions in which the reflectance behavior is equal within a tolerance range.

[0071] According to a further embodiment, in step 715 of subdividing, a region in the image is interpreted as a disturbance region in which a disturbance is located on an optical surface in a light path if the reflection behavior in the disturbance region does not change over a predefined period of time. In step 720 of changing, the power of the illumination light in the disturbance region is then reduced.

[0072] According to a further embodiment, in step 715 of subdividing, a region in the image is interpreted as a pupil region in which a pupil of one of the person's eyes is imaged if the reflection behavior in the pupil region changes from an initial value and subsequently returns to the initial value. In step 720 of changing, the power of the illumination light in the pupil region is then increased.

[0073] According to a further embodiment, in step 715 of subdividing, a region is formed as a transverse strip of the image of the person's eye area.

[0074] In step 720 of changing, a power of the illumination light in at least one of the regions is changed.

[0075] According to one embodiment, in step 725 of illumination, the person's eye area is illuminated with a homogeneous light intensity if no pupil of the person has been detected. Subsequently, for example, steps 710, 715, and 720 of reading, subdividing, and modifying are repeatedly executed. In step 710 of the repeated reading, for example, an image signal is read in that represents a person's eye area illuminated with the homogeneous light intensity.

[0076] Fig. 8 shows a block diagram of an embodiment of a device 130 for operating a device for tracking a person's gaze direction. The device 130 is configured, for example, to execute and / or control the method of Fig. 7 or a similar method.

[0077] The device 130 comprises an illumination unit 805, a scanning unit 810, a subdivision unit 815, and a modification unit 820. Optionally, the device 130 comprises an illumination unit 825.

[0078] In the illuminating unit 805, an eye area of ​​the person is illuminated with an illuminating light.

[0079] In the reading unit 810, an image signal 812 is read in, representing an image of reflections of a person's eye area. According to one embodiment, the image signal 812 represents a filter image. The filter image was determined using low-pass filtering from a plurality of images of the person's eye area. In particular, the plurality of images of the person's eye area were taken while the person had their eyes closed.

[0080] In the subdividing unit 815, the image is divided into a plurality of regions depending on the reflection behavior in the respective regions. According to one embodiment, the image is divided into regions in which the reflection behavior is equal within a tolerance range.

[0081] According to a further embodiment, in the subdividing unit 815, a region in the image is interpreted as a disturbance region in which a disturbance is located on an optical surface in a light path if the reflection behavior in the disturbance region does not change over a predefined period of time. In the modifying unit 820, the power of the illumination light in the disturbance region is then reduced.

[0082] According to a further embodiment, in the subdividing unit 815, a region in the image is interpreted as a pupil region in which a pupil of one of the person's eyes is imaged if the reflection behavior in the pupil region changes from an initial value and subsequently returns to the initial value. In the changing unit 820, the power of the illumination light in the pupil region is then increased.

[0083] According to a further embodiment, in the dividing unit 815, a region is formed as a cross strip of the image of the person's eye area.

[0084] In the changing unit 820, a power of the illumination light in at least one of the regions is changed.

[0085] According to one embodiment, the person's eye area is illuminated with a homogeneous light intensity in the illumination unit 825 if no pupil of the person has been detected. Subsequently, the reading, subdividing, and modifying units 810, 815, and 820 are repeatedly executed, for example. In the repeatedly executed reading unit 810, an image signal 814 is read, for example, that represents a person's eye area illuminated with the homogeneous light intensity.

[0086] If an embodiment comprises an “and / or” link between a first feature and a second feature, this is to be read as meaning that the embodiment according to one embodiment has both the first feature and the second feature and according to another embodiment has either only the first feature or only the second feature.

Claims

Claims 1. A method (700) for operating a device (130) for tracking a line of sight (144) of a person, the method (700) comprising the following steps: illuminating (705) an eye area of the person with an illuminating light; Reading (710) an image signal (812) representing an image (200; 300; 400) of reflections of an eye area of the person; Dividing (715) the image (200; 300; 400) into a plurality of regions (150; 500, 505, 510) depending on a reflection behavior in the respective regions (150; 500, 505, 510); and Changing (720) a power of the illumination light in at least one of the regions (150; 500, 505, 510).

2. The method (700) according to claim 1, wherein in the step (715) of dividing the image is divided into regions (150; 500, 505, 510) in which the reflection behavior is the same within a tolerance range.

3. Method (700) according to one of the preceding claims, wherein in the step (715) of subdividing, a region (505, 510) in the image (400) is interpreted as a disturbance region in which a disturbance is located on an optical surface in a light path if the reflection behavior in the disturbance region does not change over a predefined period of time, wherein in the step (720) of changing, the power of the illumination light in the disturbance region is reduced.

4. Method (700) according to one of the preceding claims, wherein in the step (715) of subdividing, a region (150; 500) in the image is interpreted as a pupil region in which a pupil (140) of an eye (135) of the person is imaged if the reflection behavior of pixels in the pupil region changes from an initial value and subsequently returns to the initial value, wherein in the step (720) of changing, the power of the illumination light in the pupil region is increased.

5. The method (700) according to any one of the preceding claims, wherein in the step (715) of dividing, a region is formed as a transverse stripe of the image (400) of the eye area of the person.

6. The method (700) according to any one of the preceding claims, wherein in the step (710) of reading in, the image signal (812) is read in, which represents a filter image determined using low-pass filtering from a plurality of images of the eye region of the person, in particular wherein the plurality of images of the eye region of the person were taken while the person had their eyes closed.

7. Method (700) according to one of the preceding claims, comprising a step (725) of illuminating the eye region of the person with a homogeneous light intensity if no pupil (140) of the person has been detected, wherein the steps (710, 715, 720) of reading, subdividing and modifying are subsequently carried out repeatedly, wherein in the step (710) of repeatedly carried out reading, an image signal (814) is read in which represents an eye region of the person illuminated with the homogeneous light intensity.

8. Device (130) which is configured to carry out and / or control the steps (705, 710, 715, 720, 725) of the method (700) according to one of the preceding claims in corresponding units (805, 810, 815, 820, 825).

9. Data glasses (100) comprising a device (130) according to claim 8.

10. Computer program configured to execute and / or control the steps of the method (700) according to one of claims 1 to 7. 11 . Machine-readable storage medium on which the computer program according to claim 10 is stored

Images