Device and method for detecting a pupil, the device comprising data glasses

Abstract

Device (102) and method for detecting a pupil, wherein a scanning area is scanned line by line with an infrared laser beam (108), wherein brightnesses of reflections (110) of the infrared laser beam (108) are detected line by line, wherein the brightest reflections in particular are determined, the brightness of which exceeds a predetermined threshold, wherein a Voroni diagram (200) is determined line by line depending on the brightnesses detected line by line, wherein the brightest reflections in particular, the brightness of which exceeds the predetermined threshold, form the origin centers of Voroni cells in the Voroni diagram, and the pupil is detected depending on the positions of the Voroni cells in the Voroni diagram, the device (102) comprising data glasses (100).

Description

State of the art

[0001] The invention relates to a device and a method for detecting a pupil and to data glasses comprising the device. Disclosure of the invention

[0002] The device and method for detecting a pupil according to the independent claims enable pupil detection with low latency.

[0003] The method for detecting a pupil involves scanning a region line by line with an infrared laser beam. Line by line, the brightness of reflections from the infrared laser beam is recorded, and the brightest reflections, whose brightness exceeds a predefined threshold, are identified. Based on the brightness measurements recorded line by line, a Voroni diagram is generated line by line. The brightest reflections, whose brightness exceeds the predefined threshold, form the origin centers of Voroni cells in the Voroni diagram. The pupil is then detected based on the positions of these Voroni cells in the Voroni diagram. By generating the Voroni diagram line by line and detecting the pupil based on it, it is not necessary to wait until the end of the entire scanning process before the pupil is detected. This reduces the detection latency.

[0004] In one example, it is provided that Voroni cells with an area smaller than a given area are assigned to the pupil, and / or that Voroni cells with an area larger than a given area are assigned to a neighborhood of the pupil.

[0005] In one example, it is proposed that the Voroni cells, depending on a classification of an area of ​​the respective Voroni cell, are assigned as belonging to the pupil or as belonging to the environment using an artificial neural network.

[0006] It may be provided that a two-dimensional, in particular binary, representation of the Voroni diagram is determined, in particular row by row, wherein the two-dimensional representation in the areas of the Voroni cells that are assigned to the pupil comprises pixels that represent the pupil, wherein the two-dimensional representation in the areas of the Voroni cells that are assigned to the environment of the pupil comprises pixels that represent the environment, wherein each pixel is assigned a first position of the respective pixel in the first dimension of the two-dimensional representation, wherein each pixel is assigned a second position of the respective pixel in the second dimension of the two-dimensional representation, wherein a two-dimensional normal distribution is determined, in particular row by row, which in each dimension comprises a normal distribution over the positions of pixels representing the pupil.The pupil is detected in the two-dimensional representation depending on the two-dimensional normal distribution. Voroni cells have a grown shape. The two-dimensional normal distribution allows for the approximate detection of an oval or round pupil.

[0007] For example, for each normal distribution, a mean and a standard deviation of the pixel positions in the respective dimension representing the pupil are determined. The mean provides a good indication of the actual position of the pupil and can be easily updated, especially row by row.

[0008] For example, the representation of the Voroni diagram is determined row by row, with the second position representing a position in the respective row. This assignment simplifies the calculation.

[0009] It can be provided that the row-by-row determination of the Voroni diagram and its representation is aborted if the row-by-row mean of the normal distribution determined for the second dimension no longer changes for at least one row. This reduces the latency, as the remainder of the sampled range is no longer evaluated.

[0010] For example, an image of the scanned area is determined, where the image comprises pixels, and where the pupil in the image is detected according to the two-dimensional normal distribution. This image and the pupil detected within it are available immediately after the last line of the image has been scanned. This reduces latency.

[0011] Estimating the position of the pupil involves assigning the pixels from the image to the pupil that lie within a confidence interval defined by the two-dimensional normal distribution around the mean of the two-dimensional normal distribution.

[0012] A device for detecting a pupil provides that the device comprises a beam deflection device for scanning with an infrared laser beam and at least one receiving unit for detecting reflections of the infrared laser beam, wherein the device is configured to carry out the method.

[0013] It may be possible to use data glasses that include the device.

[0014] Further embodiments can be found in the following description and the drawing. The drawing shows: Fig. 1. A schematic representation of data glasses with a device for detecting a pupil, Fig. 2 an exemplary Voroni diagram of a sampling range, Fig. 3. A histogram of the area of ​​the surfaces of Voroni cells from the Voroni diagram. Fig. 4 a binary representation of the detected pupil and its surroundings, Fig. 5 a part of the Voroni diagram of the range based on already sampled lines, Fig. 6 normal distributions over the binary representation, Fig. 7 an image of the scanning area with an ellipse representing the detected pupil in its surroundings, Fig. 8 A flowchart showing the steps of a procedure for detecting the pupil.

[0015] In Fig. Figure 1 is a schematic representation of a data glasses 100 with a device 102 for detecting a pupil 104.

[0016] The device 102 comprises a micromirror 106 for scanning with an infrared laser beam 108 and at least one photodiode 110 for detecting reflections of the infrared laser beam 108. In the example, the device 102 comprises an infrared laser 112 for generating the infrared laser beam 108.

[0017] The infrared laser 112 emits light, preferably at 850nm or 940nm or another wavelength in the infrared range.

[0018] The infrared laser 112 is, for example, a laser diode. The infrared laser 112 is, for example, implemented as an edge emitter or VCSEL.

[0019] The light is then deflected by the micromirror 106. The micromirror 106 is an example of a beam deflection device designed to scan a two-dimensional area with the infrared laser beam 108.

[0020] In the data glasses 100, the micromirror 106 and the infrared laser 112 are arranged in such a way that the pupil of a wearer of the data glasses 100 lies within the scanning area.

[0021] The device 102 is configured to control the micromirror 106 for line-by-line scanning of the scanning area with the infrared laser beam 108. In this example, a rectangular scanning area is scanned line by line.

[0022] The photodiode 110 is configured to detect reflections pixel by pixel. The device 102 is configured to determine pixels of a two-dimensional image of the scanning area based on the reflections. The device 102 is configured to detect the pixels of the image during line-by-line scanning.

[0023] The two-dimensional image has a first dimension and a second dimension. The first dimension represents the position of each pixel within a row. The second dimension represents the position of the row within the image.

[0024] The device 102 is designed to detect the brightness of the respective pixel and the position of the respective pixel during line-by-line scanning.

[0025] An optional projection lens can be provided between the micromirror 106 and the scanning area, which collimates or focuses the infrared laser beam 108.

[0026] The brightness of the pixel depends on whether the infrared laser beam 108 hits the sclera, cornea, iris or pupil 104.

[0027] Starting from the area of ​​the eye which is hit by the infrared laser beam 108, the infrared laser beam 108 is reflected, scattered or absorbed according to the scattering properties of the surface.

[0028] In particular, when the infrared laser beam 108 penetrates the pupil, it acts as an aperture for the scattered light, so that little light is reflected back.

[0029] In this example, backscattered light is directed via the micromirror 106 to the photodiode 110.

[0030] The photodiode 110 is an example of a receiving device.

[0031] In this example, the receiving device is integrated into the infrared laser 112. For example, the infrared laser 112 comprises a laser cavity containing the laser diode. The laser diode includes a rear laser facet with a distributed Bragg reflector (DBR). The photodiode 110 is, for example, directly integrated into the distributed Bragg reflector (DBR) of the rear laser facet of the laser diode, so that the photodiode 110 measures the optical power of the infrared laser 112. When light is reflected back into the laser cavity, this leads to a modulation (by interference) of the optical power, which is then measured by the photodiode 110.

[0032] The receiving device may be integrated into the beam path of the infrared laser beam 108, particularly via a beam splitter. Alternatively, the receiving device may be externally mounted or reach the scattered light without reflection via the micromirror 106.

[0033] Alternatively, the receiving device can be implemented as a single photon avalanche photodiode (SPAD) CCD or CMOS sensor. Alternatively, the receiving device can include a neuromorphic camera.

[0034] The infrared laser 112, for example, is controlled via a laser driver.

[0035] In this example, the laser driver is integrated into a System On Chip (SoC) 114.

[0036] The infrared laser 112, for example, is controlled by the system on chip (SoC) 114 via a control line 116.

[0037] The infrared laser 112, for example, is operated by the system-on-chip (SoC) 114 in a continuous wave (CW) mode. The infrared laser 112 is modulated by the system-on-chip (SoC) 114. The modulation can be either on-off keying (OOK) or continuous frequency modulated continuous wave (FMCW).

[0038] In this example, the micromirror 106 is controlled via a driver circuit. In this example, the driver circuit is integrated into the SoC 114.

[0039] The micromirror 106, for example, is controlled by the SoC 114 via a control line 118.

[0040] The control line 118 transmits, for example, a current angle α for the deflection in the first dimension and a current angle β for the deflection in the second dimension. This means that the position of the pixels and the position that the infrared laser 108 scans in the scanning area are correlated and known for each pixel.

[0041] The backscattered light is amplified, filtered and digitized, for example via a preamplifier and an analog-to-digital converter.

[0042] In this example, the SoC 114 includes the preamplifier and the analog-to-digital converter. In this example, the photodiode 110 is connected to the SoC 114 via a signal line 120.

[0043] A lens and / or a bandpass filter tuned to the laser wavelength can be integrated in front of the photodiode 110 in order to improve the signal transmitted via the signal line 120 by increasing photon collection efficiency or to minimize interference signals via the bandpass filter.

[0044] The device 102 is designed to perform a method for detecting the pupil.

[0045] Pupil detection is based on a Voroni diagram of the sampling range.

[0046] In Fig. Figure 2 shows an example Voroni diagram 200 over the sampling range. In this example, the sampling range is square and extends in a first direction 202 and a second direction 204. The second direction 204 represents the direction of the lines during sampling. The first direction 202 corresponds to the sequence in which the respective lines are sampled.

[0047] The Voroni diagram 200 includes Voroni cells. An example Voroni cell is shown in Fig. 2 with 206. The Voroni cells each contain an exit center. The exit centers of the Voroni cells are represented as points in Voroni diagram 200. The exit center of the example Voroni cell is in Fig. 2 labeled 208.

[0048] In Fig. 2 is an area 210 which contains more Voroni cells with a smaller area compared to the other areas than a neighborhood of area 208.

[0049] The positions of the pixels captured during scanning are uniquely assigned to one of the cells of Voroni diagram 200. In this example, Voroni diagram 200 is represented by pixels. For instance, the pixels captured during scanning and the pixels of Voroni diagram 200 are mapped one-to-one. In this example, "one-to-one mapping" means that the positions of the pixels that are mapped one-to-one are identical.

[0050] The surfaces of the Voroni cells differ from each other.

[0051] In Fig. Figure 3 shows a histogram 300 plotted against the areas 302 of Voroni cells from the Voroni diagram 200 plotted against their frequency 304. Using the histogram, Voroni cells with areas smaller than a predefined value are assigned to the pupil. Similarly, Voroni cells with areas larger than a predefined value are assigned to the area surrounding the pupil.

[0052] In Fig. Figure 4 is a binary representation 400 of the Voroni diagram 200. In this example, the binary representation 400 extends in the first direction 202 and in the second direction 204.

[0053] In Fig. Figure 4 depicts a detected pupil and its surroundings. The two-dimensional representation 400 includes, within the areas of the Voroni cells associated with the pupil, pixel 402, which represents the pupil. The two-dimensional representation 400 also includes, within the areas of the Voroni cells associated with the pupil's surroundings, pixel 404, which represents the surroundings.

[0054] Each pixel is assigned a first position of the respective pixel in the first direction 202 and a second position of the respective pixel in the second direction of the two-dimensional representation 400.

[0055] The pupil is detected in region 210 of Voroni diagram 200. In this example, the surrounding area is region 208 from Voroni diagram 200.

[0056] The Voroni diagram 200 is determined in the example by scanning the sampling range row by row.

[0057] In Fig. Figure 5 shows a portion of the Voroni diagram 200, based on already sampled lines. This portion of the Voroni diagram extends in the first direction 202 and in the second direction 204. The sampled lines extend in the second direction 204.

[0058] By scanning the lines, a new line is added to the Voroni diagram 200 in the first direction 202.

[0059] In Fig. Figure 6 shows a first normal distribution 602 and a second normal distribution 604 over the binary representation 400.

[0060] Normal distributions each have a mean and a standard deviation.

[0061] The first normal distribution 602 describes the distribution of the positions of the pixels in the first direction, which are assigned to the pupil in the binary representation 400. The first normal distribution 602 is characterized by a first mean µ1 and a first standard deviation σ1.

[0062] The second normal distribution 604 describes the distribution of the positions of the pixels in the second direction, which are assigned to the pupil in the binary representation 400. The second normal distribution 604 is characterized by a second mean µ2 and a second standard deviation σ2.

[0063] The first normal distribution and the second normal distribution are parts of a two-dimensional normal distribution over the positions of pixels representing the pupil. The two-dimensional normal distribution is characterized by the mean µ = (µ1,µ2) and standard deviation σ = (σ1, σ2).

[0064] In this example, the pixels of the two-dimensional representation 400 that lie within a predefined confidence interval around the mean of the two-dimensional normal distribution are determined to be the pixels assigned to the pupil. The confidence interval includes, for example, the pixels for which it is determined with a probability that lies within a predefined multiple of the standard deviation σ = (σ1, σ2) around the mean µ = (µ1,µ2) of the two-dimensional normal distribution that the pixels are assigned to the pupil.

[0065] In Fig. Figure 7 shows an image 700 of the sampling area with an ellipse 702. The ellipse 702 represents the confidence interval, i.e., the detected pupil within its environment 704.

[0066] In Fig. Figure 8 shows a flowchart with steps of the procedure for pupil detection.

[0067] The procedure includes step 802.

[0068] In step 802, one line of the sampling range is sampled.

[0069] In step 802, the brightness of reflections from the infrared laser beam 108 is measured. The reflections are detected, for example, using photodiode 110. The brightness of the reflections is defined, for example, by the amplitude of a signal generated by photodiode 110.

[0070] Then step 804 is executed.

[0071] In step 804, the brightest reflections are determined line by line, those whose brightness exceeds a predefined threshold. This threshold defines, for example, the amplitude at which a reflection is considered the output center of a Voroni cell.

[0072] Then step 806 is executed.

[0073] In step 806, the Voroni diagram 200 is determined line by line, depending on the brightness values ​​recorded line by line.

[0074] In the example, the brightest reflections, whose brightness exceeds the specified threshold, form the starting centers of the Voroni cells in the Voroni diagram 200.

[0075] Then, step 808 is executed.

[0076] In step 808, the pupil is detected depending on the positions of the Voroni cells in the Voroni diagram 200.

[0077] In this example, steps 802 to 808 are performed row by row. This means that the pupil is detected in step 808 using the Voroni diagram, which has been extended by the respective row.

[0078] In this example, an infrared laser beam is used to scan an area, the scanning area. Voroni cells, representing this area, are identified. The pupil is then detected based on the number of Voroni cells within this area.

[0079] In this example, the area of ​​the eye is scanned.

[0080] To move from one line to the next, the infrared laser beam is moved to the next line by changing a first angle of the micromirror 106. To scan within a line, the infrared laser beam is moved within the line by changing a second angle of the micromirror 106.

[0081] In the example, the position in the first direction 202 is defined by the first angle. In the example, the position in the second direction 204 is defined by the second angle.

[0082] The position of the output centers of the Voroni cells is determined, for example, depending on the angles at which the brightest reflections, whose brightness exceeds the specified threshold, are detected.

[0083] By specifying the threshold value, the sensitivity and granularity of the detection and the number of output centers are influenced.

[0084] In one example, the reflection is assigned to the angle of the micromirror 106 that is present at the same time as the reflection is detected. The assignment is made, for example, using the respective time.

[0085] In the example, the binary representation 400 of the pupil and the area surrounding the pupil is determined depending on the areas of the Voroni cells.

[0086] For example, Voroni cells whose area is smaller than a predefined area are assigned to the pupil. The other Voroni cells are assigned to the surrounding area, for example. This represents a rule-based evaluation.

[0087] It can be implemented that the Voroni cells are assigned to either pupil 402 or environment 404, depending on a classification of the areas using an artificial neural network. Classification with a specially trained artificial neural network further reduces latency compared to rule-based evaluation.

[0088] Depending on the two-dimensional representation 400, the two-dimensional normal distribution is determined. In the example, the two-dimensional normal distribution is determined row by row.

[0089] The two-dimensional normal distribution comprises one normal distribution per dimension over the positions of the pixels representing the pupil. In the example, the two-dimensional normal distribution includes the first normal distribution 602 and the second normal distribution 604.

[0090] In this example, the pupil is detected based on the two-dimensional normal distribution. The pixels of the two-dimensional representation 400 that lie within a predefined confidence interval around the mean of the two-dimensional normal distribution are identified as corresponding to the pupil. This confidence interval includes, for example, the pixels for which it is determined with a probability that lies within a predefined multiple of the standard deviation σ = (σ1, σ2) around the mean µ = (µ1,µ2) of the two-dimensional normal distribution that the pixels correspond to the pupil.

[0091] In this example, the assignment of Voroni cells to the pupil or surrounding area is determined after the scanning of each row is complete, based on the Voroni diagram that comprises that row. It may be possible to start the assignment process during the scanning of the row, for example, for Voroni cells that have already fully grown. Alternatively, it may be possible to maintain the assignment of previously assigned Voroni cells during the scanning process, for example, for Voroni cells that have already fully grown and will no longer change as a result of the scanning.

Claims

[1] Methods for detecting a pupil, characterized by , that an infrared laser beam (108) is used to scan a scanning area line by line, wherein brightnesses of reflections (110) of the infrared laser beam (108) are detected line by line (802), wherein in particular the brightest reflections are determined (804) whose brightness exceeds a predetermined threshold, wherein a Voroni diagram (200) is determined line by line depending on the brightnesses detected line by line (806), wherein in particular the brightest reflections whose brightness exceeds the predetermined threshold form the output centers (208) of Voroni cells (206) in the Voroni diagram (200), and the pupil is detected depending on the positions of the Voroni cells (206) in the Voroni diagram (200) (808). [2] Method according to claim 1, characterized by, that Voroni cells (210) with an area smaller than a given area of ​​the pupil (402) are assigned, and / or that Voroni cells (206) with an area larger than a given area of ​​a neighborhood (404) of the pupil are assigned (808). [3] Method according to claim 1, characterized by , that the Voroni cells are assigned, depending on a classification of an area of ​​the respective Voroni cell with an artificial neural network, as belonging to the pupil (402) or as belonging to the environment (404) (808). [4] Method according to one of claims 2 or 3, characterized by, that a two-dimensional, in particular binary, representation (400) of the Voroni diagram (200) is determined, in particular row by row (808), wherein the two-dimensional representation (400) comprises pixels (402) representing the pupil in the areas of the Voroni cells associated with the pupil, wherein the two-dimensional representation (400) comprises pixels (404) representing the pupil in the areas of the Voroni cells associated with the environment of the pupil, wherein each pixel is assigned a first position of the respective pixel in the first dimension of the two-dimensional representation, wherein each pixel is assigned a second position of the respective pixel in the second dimension of the two-dimensional representation (400), wherein a two-dimensional normal distribution is determined, in particular row by row, which comprises a normal distribution over the positions of pixels representing the pupil in each dimension,where the pupil is detected in the two-dimensional representation depending on the two-dimensional normal distribution (808). [5] Method according to claim 4, characterized by , that for each normal distribution a mean and a standard deviation of the positions of the pixels in the respective dimension that represent the pupil are determined. [6] Method according to claim 4 or 5, characterized by , that the representation of the Voroni diagram is determined row by row, with the second position representing a position of the respective row. [7] Method according to claim 6, characterized by , that the row-wise determination of the Voroni diagram and the representation is terminated if the row-wise determined mean of the normal distribution determined for the second dimension no longer changes for at least one row. [8] Method according to claims 4 to 7, characterized by, that an image of the sampling area is determined, wherein the image comprises pixels, and wherein the pupil in the image is detected depending on the two-dimensional normal distribution (808). [9] Method according to claim 8, characterized by , where the pixels from the image are assigned to the pupil which lie within a confidence interval (702) defined by the two-dimensional normal distribution around the mean (µ1,µ2) of the two-dimensional normal distribution. [10] Device (102) for detecting a pupil, characterized by , that the device (102) comprises a beam deflection device (106) for scanning with an infrared laser beam (108) and at least one receiving unit (110) for detecting reflections (110) of the infrared laser beam (108), wherein the device (102) is configured to carry out the method according to one of claims 1 to 9. [11] Data glasses (100), characterized by, that the data glasses (100) comprise the device (102) according to claim 10.

Images