Line-of-sight detection device

Abstract

Provided is a line-of-sight detection device that can minimize a reduction in detection accuracy regardless of an imaging method employed by an imaging system. A line-of-sight detection device according to the present technology comprises: a patterned light irradiation system that irradiates an eyeball with patterned light; an imaging system that captures an image of the eyeball irradiated with the patterned light; and a line-of-sight detection system that detects a line-of-sight that is an orientation of the eyeball based on the image captured by the imaging system. The patterned light has a shape conforming to a reference pattern including at least three non-parallel lines and having an intersection point at which at least two lines of the at least three lines intersect each other. By means of the present technology, it is possible to provide a line-of-sight detection device capable of minimizing a reduction in detection accuracy regardless of the type of imaging method adopted by an imaging system.

Description

Technical Field

[0001] A technology according to the present disclosure (hereinafter also referred to as “the present technology”) relates to a visual line detection device. Background Art

[0002] A conventionally known sight line detection device detects a sight line, that is, the direction of an eyeball, by illuminating the eyeball with patterned light and imaging the eyeball with an imaging system.

[0003] Conventional gaze detection devices include: a device that illuminates the eyeball with patterned light shaped like a rectangular frame (for example, see patent document 1); and a device that illuminates the eyeball with patterned light shaped like two line segments intersecting at a point (for example, see patent document 2).

[0004] [Citation List]

[0005] [Patent Document]

[0006] [Patent Document 1] JP2010-250789A

[0007] [Patent Document 2] JP2014-188322A Summary of the invention

[0008] [Technical issues]

[0009] However, conventional gaze detection devices may reduce detection accuracy depending on the imaging method of the imaging system (eg, event-based vision method).

[0010] Therefore, an object of the present technology is to provide a line of sight detection device capable of suppressing a decrease in detection accuracy regardless of an imaging method of an imaging system.

[0011] [Solution to the problem]

[0012] The present technology provides a sight line detection device, comprising: a patterned light irradiation system, the patterned light irradiation system irradiates an eyeball with patterned light;

[0013] an imaging system that captures an image of the eye illuminated with the patterned light; and

[0014] A gaze detection system that detects a gaze as a direction of an eyeball based on a captured image of an imaging system, wherein:

[0015] The patterned light has a shape along a reference pattern having at least three lines that are not parallel to each other, the reference pattern having an intersection point at which at least two of the at least three lines cross each other.

[0016] Each of the at least three lines may cross at least one of the other lines.

[0017] Each of the at least three lines may cross at least two of the other lines.

[0018] The reference pattern may have at least three intersection points that are not located on the same straight line.

[0019] At least three lines can cross each other at the same intersection point.

[0020] At least three lines may form continuous acute angles about the intersection point.

[0021] At least three lines may extend radially from the intersection point.

[0022] The reference pattern may have a plurality of patterns, each pattern including at least two lines crossing each other.

[0023] The plurality of patterns may include at least two patterns having the same shape.

[0024] The plurality of patterns may include at least two patterns having different shapes.

[0025] The patterned light irradiation system may irradiate the eyeball with a portion of the patterned light, the portion of the patterned light corresponding to a pattern in which intersections can be detected among the plurality of patterns.

[0026] The sight line detection system may include a cornea reflection image detection unit that detects feature points of a cornea reflection image of the patterned light from a captured image.

[0027] The cornea reflection image detection unit may associate the intersection point with the feature point based on information indicating an angle and / or a direction near the feature point of the cornea reflection image in the captured image.

[0028] The cornea reflection image detection unit may extract pixels whose brightness has increased in the captured image as pixels constituting the feature points.

[0029] The sight line detection system may include a motion information estimation unit which estimates at least one of a motion amount, a motion direction, and a motion speed of the eyeball based on a detection result in the cornea reflection image detection unit.

[0030] The motion information estimation unit can estimate the positions of feature points corresponding to the intersection points and not detected by the cornea reflection image detection unit based on the feature points corresponding to the intersection points and detected by the cornea reflection image detection unit, and then extract the movement direction of the eyeball from the estimation result.

[0031] The line of sight detection system may detect the line of sight based on at least a captured image of the imaging system and a detection result in the cornea reflection image detection unit.

[0032] The patterned light illumination system may include a light source that emits patterned light.

[0033] A patterned light illumination system may include: a light source; and an optical element that shapes light from the light source to generate patterned light.

[0034] The imaging system may include an event sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] [ Figure 1 ] Figure 1 A and Figure 1 B is an explanatory diagram of conventional eye tracking methods (1 and 2).

[0036] [ Figure 2 ] Figure 2 A and Figure 2 B is an explanatory diagram showing a disadvantage of Comparative Example 1 when performing eye tracking using EVS.

[0037] [ Figure 3 ] Figure 3 A and Figure 3 B is an explanatory diagram showing a disadvantage of Comparative Example 2 when performing eye tracking using EVS.

[0038] [ Figure 4 ] Figure 4 A to Figure 4 C is an explanatory diagram showing a disadvantage of Comparative Example 3 when performing eye tracking using EVS.

[0039] [ Figure 5 ] Figure 5 A shows an eyeball illuminated with patterned light from a sight line detection device according to the first embodiment of the present technology. Figure 5 B shows an EVS image of the eyeball.

[0040] [ Figure 6 ] Figure 6 It is a block diagram showing a functional example of the visual line detection device according to the first embodiment of the present technology.

[0041] [ Figure 7 ] Figure 7 A to Figure 7 C shows configuration examples 1 to 3 of the light source unit of the patterned light irradiation system of the visual line detection device according to the first embodiment of the present technology.

[0042] [ Figure 8 ] Figure 8 is a block diagram showing a functional example 1 of a cornea reflection image detection unit.

[0043] [ Fig. 9 ] Fig. 9: is a flowchart for describing the line of sight detection processing 1.

[0044] [ Fig.10 ] Fig.10 Is used to describe Fig. 9 Flowchart of the calibration steps.

[0045] [ Fig.11 ] Fig.11 A to Fig.11 C is an explanatory diagram of an operation example of the visual line detection device according to the first embodiment of the present technology.

[0046] [ Fig.12 ] Fig.12 Yes Get Fig.10 Illustration of the calibration values ​​in .

[0047] [ Fig.13 ] Fig.13 Is used to describe Fig. 9 Flowchart of the gaze detection steps.

[0048] [ Fig.14 ] Fig.14 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 1 of the first embodiment of the present technology. Fig.14 B shows an EVS image of the eyeball.

[0049] [ Fig.15 ] Fig.15 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 2 of the first embodiment of the present technology. Fig.15 B shows an EVS image of the eyeball.

[0050] [ Fig.16 ] Fig.16 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 3 of the first embodiment of the present technology. Fig.16 B shows an EVS image of the eyeball.

[0051] [ Fig.17 ] Fig.17 is a block diagram showing a functional example 2 of a cornea reflection image detection unit.

[0052] [ Fig.18 ] Fig.18 Shows Fig.17 An operation example of the corneal reflection image detection unit in FIG.

[0053] [ Fig.19 ] Fig.19 It is a flowchart used to describe the line of sight detection processing 2.

[0054] [ Fig. 20 ] Fig. 20 A shows an eyeball illuminated with patterned light from a sight line detection device according to a second embodiment of the present technology. Fig. 20 B shows an EVS image of the eyeball.

[0055] [ Fig.21 ] Fig.21 It is a block diagram showing a functional example of a visual line detection device according to a second embodiment of the present technology.

[0056] [ Fig. 22 ] Fig. 22 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 1 of the second embodiment of the present technology. Fig. 22 B shows an EVS image of the eyeball. Fig. 22 C is an explanatory diagram of a method for detecting intersection points from a captured image.

[0057] [ Fig.23 ] Fig.23 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 2 of the second embodiment of the present technology. Fig.23 B shows an EVS image of the eyeball.

[0058] [ Fig.24 ] Fig.24 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 3 of the second embodiment of the present technology. Fig.24 B shows an EVS image of the eyeball.

[0059] [ Fig.25 ] Fig.25 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 4 of the second embodiment of the present technology. Fig.25 B shows an EVS image of the eyeball.

[0060] [ Fig.26 ] Fig.26 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 5 of the second embodiment of the present technology. Fig.26 B shows an EVS image of the eyeball.

[0061] [ Fig. 27 ] Fig. 27 A shows an eyeball illuminated with patterned light from a visual line detection device according to a third embodiment of the present technology. Fig. 27 B shows an EVS image of the eyeball.

[0062] [ Fig.28 ] Fig.28It is a block diagram showing a functional example of a visual line detection device according to a third embodiment of the present technology.

[0063] [ Fig.29 ] Fig.29 is a block diagram showing a functional example 3 of a cornea reflection image detection unit.

[0064] [ Fig.30 ] Fig.30 It is a block diagram showing a functional example of a visual line detection device according to a fourth embodiment of the present technology.

[0065] [ Fig.31 ] Fig.31 is a block diagram showing a functional example 1 of a motion information estimation unit.

[0066] [ Fig.32 ] Fig.32 It is an explanatory diagram of the principle of functional example 1 of the motion information estimation unit.

[0067] [ Fig.33 ] Fig.33 is a block diagram showing a functional example 2 of a motion information estimation unit.

[0068] [ Fig.34 ] Fig.34 A and Fig.34 B is an explanatory diagram of the principle of functional example 2 of the motion information estimation unit.

[0069] [ Fig.35 ] Fig.35 is a block diagram showing a functional example 3 of a motion information estimation unit.

[0070] [ Fig.36 ] Fig.36 It is an explanatory diagram of the principle of functional example 3 of the motion information estimation unit.

[0071] [ Fig.37 ] Fig.37 is a block diagram showing a functional example of a pupil detection unit.

[0072] [ Fig.38 ] Fig.38 A and Fig.38 B shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 6 and 7 of the second embodiment of the present technology.

[0073] [ Fig.39 ] Fig.39 A to Fig.39 E shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 4 to 8 of the first embodiment of the present technology.

[0074] [ Fig.40 ] Fig.40 An eyeball illuminated with patterned light from a visual line detection device according to Modification Example 9 of the first embodiment of the present technology is shown.

[0075] [ Fig.41 ] Fig.41 A to Fig.41 C shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 10 to 12 of the first embodiment of the present technology.

[0076] [ Fig.42 ] Fig.42 A to Fig.42 C shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 13 to 15 of the first embodiment of the present technology.

[0077] [ Fig.43 ] Fig.43 Variation 1 of patterned light applicable to a sight line detection device according to the present technology and a corneal reflection image of the patterned light are shown.

[0078] [ Fig.44 ] Fig.44 Variation 2 of patterned light applicable to a sight line detection device according to the present technology and a corneal reflection image of the patterned light are shown. DETAILED DESCRIPTION

[0079] The preferred embodiments of the present technology will be described in detail with reference to the accompanying drawings. In this specification and the accompanying drawings, components having substantially the same functional configuration will be denoted by the same reference symbols, and thus repeated descriptions thereof will be omitted. The following embodiments describe representative embodiments of the present technology, and the scope of the present technology should not be narrowly interpreted based on the embodiments. Even if the present specification describes that the sight line detection device according to the present technology exhibits a variety of advantageous effects, the sight line detection device according to the present technology only needs to exhibit at least one of these advantageous effects. The advantageous effects described in the present specification are merely exemplary and are not restrictive, and other advantageous effects may also be obtained.

[0080] The description will be in the following order:

[0081] 0. Preface

[0082] 1. Sight line detection device according to the first embodiment of the present technology

[0083] 2. Sight Line Detection Device According to Modification Example 1 of First Embodiment of the Present Technology

[0084] 3. Sight Line Detection Device According to Modification Example 2 of First Embodiment of the Present Technology

[0085] 4. Sight Line Detection Device According to Modification Example 3 of First Embodiment of the Present Technology

[0086] 5. Sight line detection device according to the second embodiment of the present technology

[0087] 6. Sight Line Detection Device According to Modification Example 1 of Second Embodiment of the Present Technology

[0088] 7. Sight Line Detection Device According to Modification Example 2 of Second Embodiment of the Present Technology

[0089] 8. Visual Line Detection Device According to Modification Example 3 of Second Embodiment of the Present Technology

[0090] 9. Sight line detection device according to modification example 4 of second embodiment of present technology

[0091] 10. Visual Line Detection Device According to Modification Example 5 of Second Embodiment of the Present Technology

[0092] 11. Sight line detection device according to the third embodiment of the present technology

[0093] 12. Sight line detection device according to fourth embodiment of the present technology

[0094] 13. Sight line detection device according to modification examples 6 and 7 of the second embodiment of the present technology

[0095] 14. Sight line detection device according to modification examples 4 to 8 of the first embodiment of the present technology

[0096] 15. Visual Line Detection Device According to Modification Example 9 of First Embodiment of the Present Technology

[0097] 16. Sight line detection devices according to modified examples 10 to 12 of the first embodiment of the present technology 17. Sight line detection devices according to modified examples 13 to 15 of the first embodiment of the present technology

[0098] 18. Patterned light variant 1

[0099] 19. Patterned light variant 2

[0100] 20. Other modification examples of this technology

[0101] <0. Preface>

[0102] (Conventional Eye Tracking Method)

[0103] Conventionally known sight line detection devices (eye tracking devices) detect sight lines, that is, the direction of the eyeball by illuminating the eyeball with light (e.g., invisible light) and imaging the eyeball with an imaging system. As a first example of a conventional eye tracking method, Figure 1A polynomial fitting method for pupil and corneal reflection image is known. In this method, the relationship between the corneal reflection image of the light source and the pupil position is associated with the line of sight through calibration.

[0104] As a second example of a conventional eye tracking method, a model estimation method of pupil and corneal reflection images is known, such as Figure 1 B. In this method, the line of sight is detected by determining the cornea center from a plurality of cornea reflection images and the pupil center from a pupil image.

[0105] Although conventional eye tracking in the first and second examples can achieve high accuracy, a trade-off needs to be made between power consumption and frame rate. To solve this problem, for example, an EVS (Event-based Visual Sensor) can be used. EVS has recently attracted attention as a high-speed imaging element with low power consumption.

[0106] Unfortunately, if EVS is used for eye tracking, the robustness may be reduced. This is because the corneal reflection image (Purkinje image) is difficult to detect. Figure 2 A shows an image of an eyeball illuminated with light from a small point source, the image being captured by a normal imaging element. Figure 2 B shows an EVS image of the eyeball illuminated with light from a small point source. Figure 2 A and Figure 2 When a small point source is used, it is difficult to extract the corneal reflection image using the small point source according to the computer vision algorithm without gradation as shown in B. In addition, a large amount of external light such as sunlight enters the pupil as a point source, making it difficult to recognize the eyeball.

[0107] Figure 3 A shows an image of an eyeball illuminated with light from a large point source, captured by a normal imaging element. Figure 3 B shows an EVS image of the eyeball illuminated with light from a large point source. Figure 3 A and Figure 3 As shown in B, the light source can be enlarged (for example, a circular light source with a larger diameter can be used). In this case, the center of the circle can be the characteristic point of the circle, but it is not preferred because the radius of the circle is affected by blurring such as eye movement or corneal curvature.

[0108] The paper (Event-Based Kilohertz Eye Tracking using Coded DifferentialLighting) proposes a method to improve the detection accuracy of corneal reflection images when using EVS for eye tracking. The paper describes improving the detection accuracy of Purkinje by flashing two adjacent LEDs, but unfortunately, in order to prevent the flashing LEDs from causing an overall change in brightness, control must be performed, and the flashing frequency depends on the bandwidth of the EVS. The overall change in brightness may cause events to be triggered simultaneously, resulting in delays.

[0109] Therefore, the eyeball can be illuminated with patterned light to increase feature points, which may lead to the following problems:

[0110] Eye movements can change the way patterns are viewed.

[0111] Eye movement causes at least part of the pattern to disappear.

[0112] These will greatly reduce the robustness of the detection.

[0113] A simulation example of performing eye tracking by illuminating the eyeball with patterned light PL like a rectangular frame will be described below. Figure 4 A shows an image of an eyeball illuminated with patterned light PL as a rectangular frame, which is captured by a normal imaging element NS. Figure 4 B shows an image of an eyeball illuminated with patterned light PL like a rectangular frame while moving horizontally at a relatively high speed, the image being captured by the EVS. Figure 4 C shows an image of an eyeball illuminated with patterned light PL as a rectangular frame while moving horizontally at a relatively low speed, which is captured by EVS. Figure 4 In A, the image PL-NS of the reflected light of the patterned light PL can be clearly confirmed as a rectangular frame. Figure 4 In B, the rectangular frame appears as double lines in the image PL-EVS of the reflected light of the patterned light PL. These lines are thick, thus blurring the match between the light source and the reflected light. Figure 4 In C, the rectangular frame appears as a double line in the image PL-EVS of the reflected light of the patterned light PL, and the brightness has almost no horizontal change. Therefore, the shape of the patterned light PL is greatly deformed.

[0114] Therefore, through careful research, the inventors have developed a sight line detection device according to the present technology as the following sight line detection device (eye tracking device), which suppresses the reduction of detection accuracy regardless of the imaging method of the imaging system by designing the shape of patterned light. In particular, when EVS is used for an imaging system, the sight line detection device according to the present technology is quite effective.

[0115] <1. Sight line detection device according to first embodiment of the present technology>

[0116] <<Configuration of sight line detection device>>

[0117] Figure 5 A shows the eyeball EB illuminated with patterned light from the visual line detection device 10 according to the first embodiment of the present technology. Figure 5 B shows an EVS image of the eyeball EB. Figure 6 : is a block diagram showing a functional example of the visual line detection device 10 according to the first embodiment of the present technology.

[0118] The sight line detection device 10 is mounted on, for example, an HMD (head mounted display), and is used for eye tracking (sight line detection). Figure 6 As shown, the sight line detection device 10 includes a patterned light irradiation system 100, an imaging system 200 and a sight line detection system 300. For example, the sight line detection device 10 is supported by an HMD support structure (such as a glasses frame) mounted on the user's head. The HMD displays an image superimposed on the user's field of view. The HMD controls the display of the image based on, for example, the output of the sight line detection device 10 (the user's sight line). The support structure of the HMD may include a voice output device that provides voice information to the user.

[0119] (Patterned Light Irradiation System)

[0120] The patterned light irradiation system 100 uses patterned light PL1 (see Figure 5 A) irradiates the eyeball EB of the user. For example, the shape of the patterned light PL1 is a triangle. The patterned light irradiation system 100 includes, for example, a light source 100a and a light source driving unit 100b (light source driver) (see Figure 6 ). For example, the light source 100a emits invisible light (for example, infrared light). The light source 100a is, for example, a point source, a line source, or a surface source. As the light source 100a, for example, an LED (light emitting diode), an OLED (organic light emitting diode), or an LD (laser diode) can be used. Figure 5 In A and other drawings, reference symbol CO represents the cornea, and reference symbol PU represents the pupil.

[0121] The patterned light PL1 has a shape (e.g., a triangle) along a reference pattern BP, the reference pattern BP having at least three lines that are not parallel to each other, and the reference pattern BP having an intersection of at least two of the at least three lines. The at least three lines that constitute the reference pattern BP and are not parallel to each other include, for example, lines that constitute a part of a figure, lines that constitute a part of the outline of a figure, and lines that constitute a part of a character. The at least three lines may include straight lines and curves, or lines that are at least partially disconnected and solid lines. The patterned light PL1 may have a pattern that is exactly the same as the reference pattern BP, or may have a pattern that is almost the same as the reference pattern BP. In this case, the "pattern that is almost the same as the reference pattern BP" includes, for example, a pattern obtained by removing at least one intersection of the reference pattern BP, and a pattern obtained by replacing at least one intersection of the reference pattern BP with a graphic. In this pattern, the patterned light PL1 is completely consistent with the reference pattern BP. The shape of the reference pattern may be the shape of a light source, or the shape of an optical element that shapes the light from the light source.

[0122] Each of the at least three (e.g., three) lines of the patterned light PL1 intersects with at least one (e.g., two) of the other lines. Specifically, each of the at least three lines of the patterned light PL1 intersects with at least two (e.g., two) of the other lines. For example, the reference pattern BP has at least three intersections (e.g., three intersections) that are not on the same straight line. More specifically, the shapes of the reference pattern BP and the patterned light PL1 are each similar to a triangle (e.g., an isosceles triangle, particularly an equilateral triangle), one side of which extends almost horizontally, and the height direction is almost along the vertical direction.

[0123] Figure 7 A to Figure 7 C shows configuration examples 1 to 3 of the light source unit of the patterned light irradiation system 100 of the visual line detection device 10 according to the first embodiment of the present technology. Figure 7 In the configuration example 1 of A, the light source 100a having the same shape as the patterned light PL1 (for example, a triangle) is disposed on the eyeglass frame EF (for example, on the edge of the eyeglass frame EF). Figure 7 In configuration example 2 of B, a semi-transparent light source 100a (e.g., uLED) having the same shape (e.g., triangle) as the patterned light PL1 is disposed in a transparent glass or resin attached to the eyeglass frame EF (e.g., a transparent glass or resin fitted into the edge of the eyeglass frame EF). Figure 7In configuration example 3 of C, an optical element (e.g., a diffraction element) having the same shape as the patterned light PL1 is disposed in a transparent glass or resin attached to the eyeglass frame EF (e.g., a transparent glass or resin fitted into the edge of the eyeglass frame EF), and the optical element is irradiated with light from the light source 100a. In other words, the optical element shapes the light from the light source 100a into the shape of the patterned light PL1. Figure 7 A to Figure 7 In the configuration example of C, the glasses frame may be a rimless frame.

[0124] For example, the light source driving unit 100b includes circuit elements such as a transistor, a capacitor, and a resistor. The light source driving unit 100b generates a driving signal (eg, a pulse signal) and applies the signal to the light source 100a.

[0125] (Imaging System)

[0126] The imaging system 200 captures an image of the eyeball EB illuminated with the patterned light PL1, and outputs the captured image to the sight line detection system 300. Figure 6 As shown, the imaging system 200 is, for example, an imaging device including an imaging element 200a. In addition to the imaging element 200a, the imaging system 200 may further include: a focusing element (e.g., a focusing lens or a focusing mirror) that focuses the patterned light PL1 onto the imaging element 200a after the patterned light PL1 is emitted from the patterned light irradiation system 100 to the eyeball EB and reflected by the eyeball EB; and a signal processing unit that processes the output signal of the imaging element 200a. In addition, the imaging system 200 may include an imaging element (e.g., a general image sensor) for detecting the pupil.

[0127] The imaging element 200a is, for example, an EVS (event-based visual sensor). The EVS has a plurality of pixels in a two-dimensional array. The EVS is a visual sensor that detects the brightness change of each pixel and outputs the brightness value (pixel value) of only the pixel whose brightness has changed in combination with time information, and a visual sensor that realizes high-speed operation and low power consumption. For example, the EVS includes a light receiving circuit, an amplifier, and a comparator having a photoelectric conversion element (e.g., a photodiode having sensitivity in the infrared region) for each pixel. For example, in the EVS, the incident light is converted into an electrical signal by the light receiving circuit, amplified by the amplifier, and divided according to the threshold value of the brightness to be output as a positive change signal (positive event) and a negative change signal (negative event). The output signal of the EVS is output as a captured image through a signal processing unit. The EVS is also referred to as an "event sensor" or an "event visual sensor".

[0128] (Line of sight detection system)

[0129] The sight line detection system 300 includes, for example, a corneal reflection image detection unit 300a, a pupil detection unit 300b, and a sight line detection unit 300c. The sight line detection system 300 may also include a storage unit (eg, a memory). For example, the sight line detection system 300 is implemented by hardware such as a CPU and a chipset.

[0130] The cornea reflection image detection unit 300a detects feature points of the cornea reflection image of the patterned light PL1 from the captured image of the imaging system 200. Specifically, the cornea reflection image detection unit 300a detects the cornea reflection image of the patterned light PL1 from the captured image of the imaging system 200 through image processing (e.g., edge detection, contour extraction, and shadow detection), and extracts feature points of the cornea reflection image. In addition, the cornea reflection image detection unit 300a associates the intersection points of the reference pattern BP (the intersection points of the patterned light PL1) with the feature points of the cornea reflection image of the patterned light PL1.

[0131] Figure 8 : is a block diagram showing a functional example 1 of a cornea reflection image detection unit. Figure 8 As shown, the corneal reflection image detection unit 300a includes a feature point extraction unit 300a1 and an association unit 300a2. The feature point extraction unit 300a1 extracts feature points of the corneal reflection image of the patterned light PL1 by, for example, angle detection or DNN (deep neural network). The association unit 300a2 associates the feature points of the corneal reflection image of the patterned light PL1 with the intersection points of the reference pattern BP (the intersection points of the patterned light PL1) based on information about the angles and / or directions near the feature points in the captured image of the imaging system 200, stores the association results in a storage unit (e.g., a memory), and / or outputs the association results to the sight line detection unit 300c.

[0132] Specifically, in Figure 5 In A, when the eyeball EB moves in the horizontal direction, for example, only the pixel values ​​of the pixels whose brightness changes are output in the patterned light PL1 from the EVS, so that the EVS captures the image. Figure 5 B. As a supplementary explanation, when the eyeball EB moves in the horizontal direction, in response to the change in the brightness of the patterned light PL1, a corneal reflection image PL1i (output signal of the EVS) consisting of two oblique side portions in an inverted V shape appears in the image captured by the EVS. In this case, the feature point extraction unit 300a1 extracts the intersection of the two oblique side portions of the corneal reflection image PL1i as the feature point of the corneal reflection image of the patterned light PL1. In addition, the association unit 300a2 associates the feature point with the intersection of the patterned light PL1 based on information about the angle and / or direction (e.g., an acute angle or an upward direction) near the feature point (e.g., the intersection of the two oblique side portions).

[0133] exist Figure 5 In the example of A, the eyeball EB moves in a direction parallel to the line of the reference pattern BP (for example, the horizontal direction). When the eyeball EB moves in a direction that is not parallel to any of the three non-parallel lines of the reference pattern BP, the three non-parallel lines of the reference pattern BP (patterned light PL1) appear in the captured image of the EVS. In this case, three feature points (intersection points) can be detected in the corneal reflection image of the reference pattern BP.

[0134] For example, by using image processing (such as edge detection, contour extraction, and shadow detection), the pupil detection unit 300b extracts the pupil from the captured image of the imaging element used to detect the pupil, calculates the center position of the pupil in the coordinate system of the captured image, and stores the calculation result in a storage unit (such as a memory), and / or outputs the calculation result to the line of sight detection unit 300c.

[0135] For example, the sight line detection unit 300c detects the sight line, that is, the direction of the eyeball EB, based on the detection result obtained by the pupil detection unit 300b according to the captured image of the imaging system 200 and the detection result obtained by the cornea reflection image detection unit 300a.

[0136] <<Gaze Detection Processing 1>>

[0137] Reference Figures 9 to 13 , the following will describe the line of sight detection processing 1 performed by, for example, the line of sight detection device 10. Fig. 9 : is a flowchart for describing the line of sight detection processing 1.

[0138] In step S1, a calibration step is performed. The calibration step will be described in detail later.

[0139] In step S2, a sight line detection step is performed. The sight line detection step will be described in detail later. After executing step S2, the process terminates.

[0140] (Step S1: Calibration Step)

[0141] Reference Fig.10 The flowchart is described below. Fig. 9 The calibration steps are as follows.

[0142] In the first step S1-1, the user's sight is guided to a predetermined direction. Specifically, the user is notified (for example, through the display of the HMD or the voice of the voice output device) to look at a predetermined gaze point (for example, the front), so that the user's sight is guided to the predetermined direction (for example, the front).

[0143] In the next step S1-2, the eyeball EB is illuminated with the patterned light PL1. Specifically, the light source 100a is driven by the light source driving unit 100b.

[0144] In the next step S1-3, imaging is performed. Specifically, an image of the eyeball EB illuminated with the patterned light PL1 is captured by an imaging element (eg, a general image sensor) for detecting the pupil.

[0145] In the next step S1-4, the cornea reflection image is detected. Specifically, the cornea reflection image detection unit 300a extracts feature points of the cornea reflection image of the patterned light PL1 from the image captured in step S1-3, and associates the feature points with the intersection points of the patterned light PL1.

[0146] In the next step S1-5, the pupil is detected. Specifically, the pupil detection unit 300b extracts the pupil PU from the captured image of the imaging element for detecting the pupil, and detects the center position of the pupil.

[0147] In the next step S1-6, the calibration value is obtained. Specifically, the positional relationship between the center of the pupil and the corneal reflection image of the intersection of the patterned light PL1 detected in step S1-5 (see Fig.12 ) is associated with the sight direction (eg, forward) and is acquired as a calibration value (the cornea reflection image is detected in step S1-4), and the calibration value is then stored in a storage unit (eg, a memory).

[0148] By performing a series of steps S1-1 to S1-6 several times to obtain multiple calibration values ​​and using the average or median of the calibration values, the accuracy of the calibration values ​​can be improved. In this case, the predetermined direction in step S1-1 each time can be the same direction or a different direction.

[0149] (Step S2: Line of sight detection step)

[0150] Reference Fig.13 The flowchart is described below. Fig. 9 The sight line detection steps.

[0151] In the first step S2-1, the eyeball EB is illuminated with the patterned light PL1. Specifically, the light source 100a is driven by the light source driving unit 100b.

[0152] In the next step S2-2, imaging is performed. Specifically, the imaging element 200a (EVS) of the imaging system 200 captures an image of the eyeball EB irradiated with the patterned light PL1, and outputs the captured image (see Fig.11 B and Fig.11 C). Fig.11A shows an image of the eyeball EB illuminated with patterned light PL1, which is captured by a normal imaging element NS. Fig.11 B shows an EVS image of the eyeball illuminated with the patterned light PL1 when the eyeball moves horizontally at a relatively high speed. Fig.11 C shows an EVS image of the eyeball illuminated with patterned light PL1 when the eyeball moves horizontally at a relatively low speed. Fig.11 In A, the image PL1-NS of the reflected light of the patterned light PL1 can be clearly confirmed. Fig.11 In B, although the two oblique edges appear roughly as double lines in the image PL1-EVS of the reflected light of the patterned light PL1, the angle formed by the two oblique edges and their intersection can always be detected, so that the eyeball EB moving at a high speed can be robustly detected. Fig.11 In C, although the two oblique edges appear roughly as double lines in the image PL1-EVS of the reflected light of the patterned light PL1 and the brightness hardly changes in the horizontal direction, the angle formed by the two oblique edges and their intersection can always be detected, thereby enabling robust detection of the eyeball EB moving at a slow speed.

[0153] In the next step S2 - 3 , the cornea reflection image is detected. Specifically, the cornea reflection image detection unit 300 a extracts feature points of the cornea reflection image of the patterned light PL1 , and associates the feature points with the intersection points of the patterned light PL1 .

[0154] In the next step S2-4, the pupil is detected. Specifically, the pupil detection unit 300b extracts the pupil PU from the captured image of the imaging element for detecting the pupil, and detects the center position of the pupil.

[0155] In the next step S2-5, the line of sight is detected based on the calibration value. Specifically, the line of sight direction is determined based on the positional relationship between the pupil center detected in step S2-4 and the corneal reflection image of each intersection of the patterned light PL1 (the corneal reflection image was detected in step S2-3), and the calibration value acquired in the calibration step (step S1) and stored in the storage unit.

[0156] In the last step S2-6, it is determined whether to terminate the sight line detection. When the sight line detection device 10 is turned off, the termination is confirmed. When the sight line detection device 10 is turned on, the termination is rejected to perform the sight line detection again (steps S2-1 to S2-5).

[0157] <<Effect of the sight line detection device>>

[0158] The effects of the visual line detection device 10 will be described below.

[0159] The sight line detection device 10 according to the first embodiment of the present technology includes: a patterned light irradiation system 100 that irradiates an eyeball EB with patterned light PL1; an imaging system 200 that captures an image of the eyeball EB irradiated with the patterned light PL1; and a sight line detection system 300 that detects a sight line as a direction of the eyeball EB based on the captured image of the imaging system 200. The patterned light PL1 has a shape along a reference pattern BP, the reference pattern BP has at least three lines that are not parallel to each other, and the reference pattern BP has an intersection of at least two of the at least three lines.

[0160] In this case, for example, if the imaging method of the imaging system 200 is an event-based vision method, regardless of the movement direction of the eyeball EB illuminated with the patterned light PL1, at least two non-parallel lines of the patterned light PL1 can be detected in the captured image, so that feature points (such as intersection points) formed by the at least two lines can be extracted.

[0161] Therefore, according to the sight line detection device 10, regardless of the imaging method of the imaging system 200, a sight line detection device capable of suppressing a decrease in detection accuracy can be provided. If the imaging method of the imaging system 200 is not an event-based visual method (for example, in the case of a general image sensor), the sight line detection device 10 can detect at least three non-parallel lines of the patterned light PL1 in the captured image, thereby extracting a feature point (for example, an intersection point) formed by the at least three lines. In this case as well, a sight line detection device capable of suppressing a decrease in detection accuracy can be provided.

[0162] Each of the at least three (e.g., three) lines of the reference pattern BP crosses at least one (e.g., two) of the other lines. Therefore, the patterned light PL1 can have a relatively simple shape that can detect feature points regardless of the imaging method. Therefore, the light source and optical elements for generating the patterned light PL1 can have relatively simple shapes.

[0163] Each of the at least three (e.g., three) lines of the reference pattern BP crosses at least two of the other lines. Therefore, the patterned light PL1 can have a simple shape that can detect feature points regardless of the imaging method. Therefore, the light source and optical element for generating the patterned light PL1 can have a simple shape.

[0164] The reference pattern BP has at least three intersections that are not on the same straight line. Therefore, regardless of the movement direction of the eyeball EB illuminated by the patterned light PL1, at least one feature point (eg, one intersection) of the patterned light PL1 can be detected in the captured image, thereby improving detection accuracy.

[0165] The sight line detection system 300 includes a cornea reflection image detection unit 300a that detects feature points (eg, intersection points) of the cornea reflection image of the patterned light PL1 from the captured image of the imaging system 200. This enables robust detection of the movement of the eyeball EB.

[0166] The cornea reflection image detection unit 300a associates the intersection point of the reference pattern BP with the feature point based on information about the angle and / or direction near the feature point (e.g., the intersection point of the reference pattern BP) of the cornea reflection image of the reference pattern BP in the captured image of the imaging system 200. In this way, the intersection point and the feature point can be associated with high accuracy.

[0167] The sight line detection system 300 detects the sight line based on the captured image of the imaging system 200 and the detection result of the cornea reflection image detection unit 300a. This makes it possible to detect the sight line with high accuracy.

[0168] The patterned light irradiation system 100 may include a light source 100a that emits patterned light. Therefore, the light source 100a may have a simple shape, thereby being easy to manufacture.

[0169] The patterned light irradiation system 100 may include a light source 100a and an optical element that shapes light from the light source 100a to generate patterned light PL1. Therefore, the optical element may have a simple shape, thereby being easy to manufacture.

[0170] The imaging system 200 includes an EVS. This enables high-speed imaging with low power consumption.

[0171] <2. Visual Line Detection Device According to Modification Example 1 of First Embodiment of the Present Technology>

[0172] Fig.14 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 1 of the first embodiment of the present technology. Fig.14 B shows an EVS image of the eyeball.

[0173] like Fig.14 A and Fig.14 As shown in FIG. 8B , the line of sight detection device according to modification example 1 of the first embodiment has the same configuration as the line of sight detection device 10 according to the first embodiment, except that the patterned light PL2 emitted from the patterned light irradiation system 100 to the eyeball EB has a shape different from that of the patterned light PL1.

[0174] In Modification Example 1, the reference pattern BP has at least three non-parallel lines (e.g., three lines) that intersect at the same intersection and extend radially from the intersection. Specifically, in Modification Example 1, the reference pattern BP has a horizontally oriented Y-shape (a Y-shape rotated 90°). In this pattern, the patterned light PL2 is completely consistent with the reference pattern BP.

[0175] Also in Modification Example 1, for example, when the eyeball EB moves in the horizontal direction (lateral direction), the corneal reflection image PL2i of the patterned light PL2 does not have any horizontal line with constant brightness in the EVS image, but has two oblique lines with varying brightness. Extracting the intersection of these two oblique lines as a feature point enables highly accurate line of sight detection.

[0176] According to the sight line detection device of Modified Example 1, the same effect as the sight line detection device 10 according to the first embodiment can be obtained, and the reference pattern BP has a single intersection point (the intersection point of the patterned light PL2). Therefore, the corneal reflection image and the intersection point can be easily associated with each other, thereby enabling sight line detection at a higher speed.

[0177] <3. Visual Line Detection Device According to Modification Example 2 of First Embodiment of the Present Technology>

[0178] Fig.15 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 2 of the first embodiment of the present technology. Fig.15 B shows an EVS image of the eyeball.

[0179] like Fig.15 A and Fig.15 As shown in FIG. 2B , the line of sight detection device according to modification example 2 of the first embodiment has the same configuration as the line of sight detection device 10 according to the first embodiment, except that the patterned light PL3 ​​emitted from the patterned light irradiation system 100 to the eyeball EB has a shape different from that of the patterned light PL1.

[0180] In Modification Example 2, the reference pattern BP has at least three non-parallel lines that intersect at the same intersection and form continuous acute angles around the intersection. For example, in Modification Example 2, the shape of the reference pattern BP is similar to a horizontally oriented arrow. In this pattern, the patterned light PL3 ​​is completely consistent with the reference pattern BP.

[0181] Also in Modification Example 2, for example, when the eyeball EB moves in the horizontal direction, the corneal reflection image PL3i of the patterned light PL3 ​​does not have any horizontal line with constant brightness in the EVS image, but has two oblique lines with varying brightness. Extracting the intersection of these two oblique lines as a feature point enables highly accurate line of sight detection.

[0182] According to the sight line detection device of Modified Example 2, the same effect as the sight line detection device 10 according to the first embodiment can be obtained, and the reference pattern BP has a single intersection point (the intersection point of the patterned light PL3). Therefore, the corneal reflection image and the intersection point can be easily associated with each other, thereby enabling sight line detection at a higher speed.

[0183] <4. Visual Line Detection Device According to Modification Example 3 of First Embodiment of the Present Technology>

[0184] <<Configuration of sight line detection device>>

[0185] Fig.16 A shows an eyeball irradiated with patterned light PL5 from a visual line detection device according to Modification Example 3 of the first embodiment of the present technology. Fig.16 B shows an EVS image of the eyeball.

[0186] like Fig.16 A and Fig.16 As shown in FIG. 3B , the line of sight detection device according to modification example 3 of the first embodiment has the same configuration as the line of sight detection device 10 according to the first embodiment, except that the patterned light PL5 emitted from the patterned light irradiation system 100 to the eyeball EB has a shape different from that of the patterned light PL1.

[0187] In Modification Example 3, the reference pattern BP has at least three (e.g., four) non-parallel lines, each of which crosses at least two (e.g., two) of the other lines. For example, in Modification Example 3, the shape of the reference pattern BP is similar to a quadrilateral having four non-parallel sides, one of which is almost parallel to the horizontal direction. For example, it is assumed that the size of the reference pattern BP is greater than or equal to the size of the pupil PU.

[0188] In this case, the patterned light PL5 completely matches the reference pattern BP.

[0189] For example, in Modification Example 3, when the eyeball EB moves in the horizontal direction, the corneal reflection image PL5i of the patterned light PL5 does not have any horizontal line with constant brightness in the EVS image, but has three oblique lines with varying brightness (see Fig.16 B). Extracting the two intersection points formed by the three oblique lines as feature points enables highly accurate line of sight detection.

[0190] Fig.17 is a block diagram showing a functional example 2 of a corneal reflection image detection unit. In a sight line detection device according to modification example 3, as Fig.17 As shown, the cornea reflection image detection unit 300a includes a feature point extraction unit 300a1, an association unit 300a2 and a cornea center estimation unit 300a3.

[0191] Fig.18 Shows Fig.17 An operation example of the corneal reflection image detection unit. The feature point extraction unit 300a1 extracts a plurality of feature points of the corneal reflection image PL5i of the patterned light PL5 from the EVS image. The association unit 300a2 associates the feature points extracted by the feature point extraction unit 300a1 with the intersection points of the patterned light PL5. Based on the plurality of feature points extracted by the feature point extraction unit 300a1 and associated with the plurality of intersection points by the association unit 300a2, the corneal center estimation unit 300a3 determines the reflection points on the cornea of ​​the plurality of corresponding intersection points of the patterned light PL5, determines the normal vectors passing through the corresponding reflection points, estimates the intersection point of the normal vectors as the corneal center (the center of the cornea CO) in the coordinate system of the captured image, and outputs the estimation result to the line of sight detection unit 300c.

[0192] For example, by using image processing (such as edge detection, contour extraction, and shadow detection), the pupil detection unit 300b extracts the pupil from the captured image of the imaging element used to detect the pupil, calculates the center position of the pupil in the coordinate system of the captured image, and outputs the calculation result to the line of sight detection unit 300c.

[0193] The visual line detection unit 300c determines the visual line direction based on the cornea center which is the estimation result of the cornea center estimation unit 300a3 and the pupil center which is the detection result of the pupil detection unit 300b.

[0194] <<Gaze Detection Processing 2>>

[0195] Reference Fig.19 , the following will describe the line of sight detection processing 2 performed by, for example, a line of sight detection device according to modification example 3. Fig.19 It is a flowchart used to describe the line of sight detection processing 2.

[0196] In the first step S11, the eyeball EB is illuminated with patterned light. Specifically, the light source 100a is driven by the light source driving unit 100b.

[0197] In the next step S12, imaging is performed. Specifically, the imaging element 200a (EVS) of the imaging system 200 captures an image of the eyeball EB irradiated with the patterned light and outputs the captured image.

[0198] In the next step S13, the cornea reflection image is detected. Specifically, the cornea reflection image detection unit 300a extracts a plurality of feature points of the cornea reflection image of the patterned light from the captured image, and associates the cornea reflection image with the intersection points of the patterned light.

[0199] In the next step S14, the cornea center is estimated. Specifically, the cornea reflection image detection unit 300a calculates the cornea center of the cornea CO based on the feature points of the cornea reflection image of the patterned light.

[0200] In the next step S15, the pupil is detected. Specifically, the pupil detection unit 300b extracts the pupil PU from the captured image of the imaging element for detecting the pupil, and detects the center position of the pupil.

[0201] In the next step S16, the line of sight is detected. Specifically, the line of sight direction is determined based on the cornea center estimated in step S14 and the pupil center detected in step S15.

[0202] In the final step S17, it is determined whether to terminate the visual line detection. When the visual line detection device according to Modification Example 3 is off, termination is confirmed. When the visual line detection device is on, termination is rejected to perform the visual line detection (steps S11 to S16) again.

[0203] <<Effect of the sight line detection device>>

[0204] According to the line of sight detection device of Modification Example 3, at least two feature points of the corneal reflection image of the patterned light PL5 can be always detected, so that the line of sight can be detected by the line of sight detection process 2 without calibration. In addition, the patterned light PL5 is relatively large, so that the feature points of the corneal reflection image are easily detected.

[0205] <5. Sight line detection device according to second embodiment of the present technology>

[0206] Fig. 20 A shows an eyeball irradiated with patterned light PL1 from the visual line detection device according to the second embodiment of the present technology. Fig. 20 B shows an EVS image of the eyeball. Fig.21 : is a block diagram showing a functional example of the visual line detection device 20 according to the second embodiment of the present technology.

[0207] The line of sight detection device 20 according to the second embodiment has the same configuration as the line of sight detection device 10 according to the first embodiment, except that the reference pattern BP has multiple (e.g., two) patterns (e.g., triangular patterns), each pattern including at least two (e.g., three) lines crossing each other.

[0208] In the sight line detection device 20, the reference pattern BP includes at least two (for example, two) patterns of the same shape (including similar patterns, the same below) and oriented in the same direction. In this case, the patterned light corresponding to the reference pattern BP is completely consistent with the reference pattern BP. Specifically, the patterned light corresponding to the reference pattern BP includes, for example, a plurality of (for example, two) first pattern portions PL1A and a second pattern portion PL1B of the same shape and oriented in the same direction, such as Fig. 20 A. In this case, the pattern of the reference pattern BP and the pattern portion of the patterned light have a shape of a triangle in which one side extends almost horizontally.

[0209] like Fig. 20 B, for example, when the eyeball EB moves horizontally, the cornea reflection image PL1Ai of the first pattern portion PL1A and the cornea reflection image PL1Bi of the second pattern portion PL1B appear in the EVS image. The cornea reflection images each have an inverted V shape with one intersection.

[0210] In the sight line detection device 20, as Fig.21 As shown, the patterned light irradiation system 100 includes a first light source 100a1 and a second light source 100a2. The first light source 100a1 is used to generate a first pattern portion PL1A. The second light source 100a2 is used to generate a second pattern portion PL1B. The first light source 100a1 and the second light source 100a2 are driven by a light source driving unit 100b.

[0211] Furthermore, in the sight line detection device 20, sight line detection processing 2 is performed. At this time, feature points of the corneal reflection image PL1Ai (intersection points of the first pattern portion PL1A) and feature points of the corneal reflection image PL1Bi (intersection points of the second pattern portion PL1B) are extracted, and then the corneal center is calculated using the feature points.

[0212] According to the visual line detection device 20, at least two feature points of the corneal reflection image of the patterned light can be always detected, so that the visual line can be detected through the visual line detection process 2 without calibration.

[0213] In the line of sight detection device 20, the patterned light irradiation system 100 can illuminate the eyeball EB with only a portion of the patterned light (the pattern portion in which the feature point can be detected), which corresponds to the pattern in which the intersection can be detected in the multiple patterns of the reference pattern BP. As a supplementary note, when the total amount of light emission from the light source unit remains unchanged, the irradiation of the patterned light divided into multiple pattern portions may cause the brightness of each pattern portion to decrease, thereby causing the contrast to decrease and making it difficult to obtain a signal. Therefore, by reducing the number of divisions of the patterned light (when the patterned light is not divided, the number of divisions is 1), the contrast can be improved and the signal can be easily obtained. Whether the intersection can be detected in the multiple patterns of the reference pattern BP can be determined by the estimation result of the motion information estimation unit 300d (for example, the direction of movement of the eyeball EB), which will be described later. The above-mentioned selective irradiation method of patterned light is also applicable to line of sight detection devices according to other embodiments and modified examples.

[0214] <6. Visual Line Detection Device According to Modification Example 1 of Second Embodiment of Present Technology>

[0215] Fig. 22 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 1 of the second embodiment of the present technology. Fig. 22 B shows an EVS image of the eyeball. Fig. 22 C is an explanatory diagram of a method for detecting intersection points from a captured image.

[0216] The line of sight detection device according to modified example 1 of the second embodiment has the same configuration as the line of sight detection device 20 according to the second embodiment, except that the patterned light emitted to the eyeball EB has a first pattern portion PL2A and a second pattern portion PL2B, each pattern portion having a horizontally oriented Y-shape (a Y-shape rotated 90°).

[0217] like Fig. 22 B, for example, when the eyeball EB moves horizontally, the cornea reflection image PL2Ai of the first pattern portion PL2A and the cornea reflection image PL2Bi of the second pattern portion PL2B appear in the EVS image. The cornea reflection images each have a horizontally oriented V-shape with one intersection (V-shape rotated 90°).

[0218] Also in the line of sight detection device according to the modified example 1 of the second embodiment, line of sight detection processing 2 is performed. At this time, the cornea reflection image detection unit 300a extracts the feature points of the cornea reflection image PL2Ai (the intersection points of the first pattern portion PL2A) and the feature points of the cornea reflection image PL2Bi (the intersection points of the second pattern portion PL2B), and calculates the cornea center by using the feature points.

[0219] According to the sight line detection device of the modified example 1 of the second embodiment, at least two feature points of the corneal reflection image of the patterned light can be always detected, so that the sight line can be detected by the sight line detection process 2 without calibration. Even when the pattern portion is relatively small, a large acute angle can be obtained, so that the feature point (such as the intersection point) can be easily detected (see Fig. 22 C).

[0220] <7. Visual Line Detection Device According to Modification Example 2 of Second Embodiment of the Present Technology>

[0221] Fig.23 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 2 of the second embodiment of the present technology. Fig.23 B shows an EVS image of the eyeball.

[0222] like Fig.23 As shown in FIG. 1A, the sight line detection device according to the modified example 2 of the second embodiment has the same configuration as the sight line detection device 20 according to the second embodiment, except that the patterned light emitted to the eyeball EB has a first pattern portion PL1A and a second pattern portion PL1' of different shapes. The shape of the first pattern portion PL1A is a triangle with one side extending almost horizontally. The shape of the second pattern portion PL1' is similar to a triangle that is not similar to the first pattern portion PL1A and all its sides are not parallel to any side of the first pattern portion PL1A.

[0223] like Fig.23 B, for example, when the eyeball EB moves horizontally, the cornea reflection image PL1Ai of the first pattern portion PL1A and the cornea reflection image PL1'i of the second pattern portion PL1' appear in the EVS image. The cornea reflection image PL1Ai has an inverted V shape, and the shape of the cornea reflection image PL1'i is a triangle (almost the same as the shape of the second pattern portion PL1').

[0224] Also in the line of sight detection device according to the modified example 2 of the second embodiment, line of sight detection processing 2 is performed. At this time, the cornea reflection image detection unit 300a extracts the feature points of the cornea reflection image PL1Ai (the intersection points of the first pattern portion PL1A) and the feature points of the cornea reflection image PL1' (the intersection points of the second pattern portion PL1'), and calculates the cornea center by using the feature points.

[0225] According to the line of sight detection device of modified example 2 of the second embodiment, at least two feature points of the corneal reflection image of the patterned light can be always detected, so that the line of sight can be detected by the line of sight detection process 2 without calibration. In addition, multiple feature points of the corneal reflection image can be detected, thereby achieving higher accuracy of line of sight detection.

[0226] <8. Visual Line Detection Device According to Modification Example 3 of Second Embodiment of the Present Technology>

[0227] Fig.24 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 3 of the second embodiment of the present technology. Fig.24 B shows an EVS image of the eyeball.

[0228] like Fig.24 As shown in A, the sight line detection device according to the modified example 3 of the second embodiment has the same configuration as the sight line detection device 20 according to the second embodiment, except that the patterned light emitted to the eyeball EB has a first pattern portion PL2A and a second pattern portion PL2B of the same shape and oriented in different directions. The first pattern portion PL2A has a Y-shape rotated at an acute angle or an obtuse angle (for example, an obtuse angle). The second pattern portion PL2B has a horizontally oriented Y-shape (a Y-shape rotated 90°). In other words, the second pattern portion PL2B has three straight lines constituting a Y-shape, so that these straight lines are not parallel to any of the three straight lines constituting the Y-shape of the first pattern portion PL2A.

[0229] like Fig.24 B, for example, when the eyeball EB moves horizontally, the cornea reflection image PL2Ai of the first pattern portion PL2A and the cornea reflection image PL2Bi of the second pattern portion PL2B appear in the EVS image. The cornea reflection image PL1Ai has a Y-shape rotated at an obtuse angle (the shape is almost the same as the first pattern portion PL2A). The cornea reflection image PL2Bi has a horizontally oriented V-shape (a V-shape rotated 90°).

[0230] Likewise, in the line of sight detection device according to the modified example 3 of the second embodiment, the line of sight detection process 2 is performed. At this time, the cornea reflection image detection unit 300a extracts the feature points of the cornea reflection image PL2Ai (the intersection points of the first pattern portion PL2A) and the feature points of the cornea reflection image PL2Bi (the intersection points of the second pattern portion PL2B), and calculates the cornea center by using the feature points.

[0231] According to the visual line detection device of Modification Example 3 of the second embodiment, at least two feature points of the cornea reflection image of the patterned light can be always detected, so that the visual line can be detected by the visual line detection process 2 without calibration.

[0232] <9. Visual Line Detection Device According to Modification Example 4 of Second Embodiment of the Present Technology>

[0233] Fig.25A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 4 of the second embodiment of the present technology. Fig.25 B shows an EVS image of the eyeball.

[0234] like Fig.25 As shown in A, the sight line detection device according to the modified example 4 of the second embodiment has the same configuration as the sight line detection device 20 according to the second embodiment, except that the patterned light emitted to the eyeball EB has a first pattern portion PL4A and a second pattern portion PL4B of the same shape and oriented in different directions. The first pattern portion PL4A has a V-shape rotated at an acute angle or an obtuse angle (e.g., an obtuse angle). The second pattern portion PL4B has a V-shape rotated at an acute angle or an obtuse angle (e.g., an obtuse angle) and oriented in a direction different from that of the first pattern portion PL4A. In other words, the second pattern portion PL4B has two straight lines constituting the V-shape, so that these straight lines are not parallel to any of the two straight lines constituting the V-shape of the first pattern portion PL4A.

[0235] like Fig.25 B, for example, when the eyeball EB moves horizontally, the cornea reflection image PL4Ai of the first pattern portion PL4A and the cornea reflection image PL4Bi of the second pattern portion PL4B appear in the EVS image. The shape of the cornea reflection image PL4Ai is almost the same as that of the first pattern portion PL4A. The shape of the cornea reflection image PL4Bi is almost the same as that of the second pattern portion PL4B.

[0236] In the line of sight detection device according to the modification example 4 of the second embodiment, the line of sight detection process 1 is performed. At this time, the cornea reflection image detection unit 300a extracts the feature points of the cornea reflection image PL4Ai (the intersection points of the first pattern portion PL4A) and / or the feature points of the cornea reflection image PL4Bi (the intersection points of the second pattern portion PL4B).

[0237] According to the visual line detection device of Modification Example 4 of the second embodiment, even when the shape of the patterned light is quite simple, at least one feature point of the cornea reflection image can be always extracted, so that the visual line can be always detected.

[0238] <10. Visual Line Detection Device According to Modification Example 5 of Second Embodiment of the Present Technology>

[0239] Fig.26 A shows an eyeball irradiated with patterned light from a visual line detection device according to Modification Example 5 of the second embodiment of the present technology. Fig.26 B shows an EVS image of the eyeball.

[0240] like Fig.26As shown in FIG. 1A , the sight line detection device according to the modified example 5 of the second embodiment has the same configuration as the sight line detection device 20 according to the second embodiment, except that the patterned light emitted to the eyeball EB has a first pattern portion PL1A and a second pattern portion PL4A with different shapes. The first pattern portion PL1A is shaped almost like a triangle with one side extending almost horizontally. The second pattern portion PL4A has an inverted V shape. The line of the second pattern portion PL4A is not parallel to any of the three sides of the first pattern portion PL1A.

[0241] For example, Fig.26 As shown in FIG. 1B, when the eyeball EB moves horizontally, the corneal reflection image PL1Ai of the first pattern portion PL1A and the corneal reflection image PL4Ai of the second pattern portion PL4A appear in the EVS image. The shape of the corneal reflection image PL1Ai is similar to the first pattern portion PL1A with a missing horizontal line. The shape of the corneal reflection image PL4Ai is almost the same as that of the second pattern portion PL4A.

[0242] Also in the line of sight detection device according to the modified example 5 of the second embodiment, the line of sight detection process 2 is performed. At this time, the cornea reflection image detection unit 300a extracts the feature points of the cornea reflection image PL1Ai (the intersection points of the first pattern portion PL1A) and the feature points of the cornea reflection image PL4Ai (the intersection points of the second pattern portion PL4A), and calculates the cornea center by using the feature points.

[0243] According to the visual line detection device of Modification Example 5 of the second embodiment, at least two feature points of the cornea reflection image of the patterned light can be always detected, so that the visual line can be detected by the visual line detection process 2 without calibration.

[0244] <11. Sight line detection device according to third embodiment of the present technology>

[0245] Fig. 27 A shows an eyeball illuminated with patterned light from a visual line detection device according to a third embodiment of the present technology. Fig. 27 B shows an EVS image of the eyeball. Fig.28 : is a block diagram showing a functional example of the visual line detection device 30 according to the third embodiment of the present technology.

[0246] The line of sight detection device 30 according to the third embodiment has the same configuration as the line of sight detection device 10 according to the first embodiment, except that the reference pattern BP has multiple (e.g., three) patterns, each pattern including at least two (e.g., two) lines intersecting each other.

[0247] In the sight line detection device 30, the reference pattern BP includes at least two (for example, three) patterns having the same shape and being oriented in different directions. In this case, the patterned light corresponding to the reference pattern BP is completely consistent with the reference pattern BP. Specifically, for example, the patterned light corresponding to the reference pattern BP includes a plurality of (for example, three) first to third pattern portions PL4A, PL4B, and PL4C having the same shape and being oriented in different directions, such as Fig. 27 A. In this case, the pattern of the reference pattern BP and the pattern portion of the patterned light are almost V-shaped. Specifically, for example, the two straight lines constituting the V-shape of each pattern portion are not parallel to any of the two straight lines constituting the V-shape of another pattern portion. The third pattern portion PL4C has an edge extending almost horizontally.

[0248] like Fig. 27 B, for example, when the eyeball EB moves horizontally, the corneal reflection image PL4Ai of the first pattern portion PL4A, the corneal reflection image PL4Bi of the second pattern portion PL4B, and the corneal reflection image PL4Ci of the third pattern portion PL4C appear in the EVS image. The corneal reflection images PL4Ai and PL4Bi are each V-shaped with one intersection. The corneal reflection image PL4Ci is a straight line.

[0249] In the sight line detection device 30, as Fig.28 As shown, the patterned light irradiation system 100 includes first to third light sources 100a1, 100a2 and 100a3. The first light source 100a1 is used to generate a first pattern portion PL4A. The second light source 100a2 is used to generate a second pattern portion PL4B. The third light source 100a3 is used to generate a third pattern portion PL4C. The first to third light sources 100a1, 100a2 and 100a3 are driven by a light source driving unit 100b.

[0250] Also in the line of sight detection device 30, line of sight detection processing 2 is performed. At this time, feature points of the corneal reflection image PL4Ai (intersection points of the first pattern portion PL4A), feature points of the corneal reflection image PL4Bi (intersection points of the second pattern portion PL4B), and feature points of the corneal reflection image PL4Ci (intersection points of the third pattern portion PL4C) are extracted, and the corneal center is calculated using the feature points.

[0251] According to the visual line detection device 30, at least two feature points of the corneal reflection image of the patterned light can be always detected, so that the visual line can be detected through the visual line detection process 2 without calibration.

[0252] (Example of a method for detecting a cornea reflection image detection unit)

[0253] In the aforementioned embodiment and modification examples, the cornea reflection image detection unit 300 a may extract pixels with increased brightness in the EVS image as feature points. Fig.29 is a block diagram showing a functional example 3 of a cornea reflection image detection unit. Fig.29 As shown, the functional example 3 of the corneal reflection image detection unit 300a includes a brightness comparison unit 300a4 in addition to the feature point extraction unit 300a1 and the association unit 300a2. The brightness comparison unit 300a4 compares the brightness of the pixels in the current frame with the brightness of the pixels in the previous frame, which constitute the feature points extracted by the feature point extraction unit 300a1 and associated by the association unit 300a2. Then, the brightness comparison unit 300a4 extracts only the pixels with increased brightness from the pixels as the pixels constituting the feature points. Therefore, the feature points can be extracted more strictly.

[0254] <12. Sight line detection device according to fourth embodiment of the present technology>

[0255] Fig.30 : is a block diagram showing a functional example of the visual line detection device 40 according to the fourth embodiment of the present technology.

[0256] like Fig.30 As shown, the visual line detection device 40 according to the fourth embodiment has almost the same configuration as the visual line detection device 10 according to the first embodiment, except that the visual line detection system 300 further includes a motion information estimation unit 300d.

[0257] The movement information estimation unit 300d estimates at least one of the movement amount, movement direction, and movement speed of the eyeball EB based on the detection result in the cornea reflection image detection unit 300a.

[0258] (Function Example 1 of Motion Information Estimation Unit)

[0259] Fig.31 : is a block diagram showing a functional example 1 of the motion information estimation unit 300 d . Fig.31 The motion information estimation unit 300d in the embodiment includes a motion amount estimation unit 300d1 and a motion direction estimation unit 300d2. The motion amount estimation unit 300d1 can extract, for example, a feature point with reduced brightness and a feature point with increased brightness, and estimate the motion amount of the eyeball EB based on the positional relationship between the two feature points. The motion direction estimation unit 300d2 can estimate the motion direction of the eyeball EB based on the positional relationship, or estimate the motion direction of the eyeball EB based on, for example, the direction of the acute angle formed by two straight lines intersecting at the feature point (intersection point) of the corneal reflection image PL4i or the event polarity (positive or negative), such as Fig.32 shown.

[0260] In the sight line detection device 40, the sight line detection unit 300c performs the sight line detection process 1. At this time, in the calibration step and / or the sight line detection step, in addition to the feature points of the corneal reflection image and the center of the pupil, the movement amount and movement direction of the eyeball EB are also associated with the sight line, and the movement amount and movement direction are estimated by the motion information estimation unit 300d, so that the sight line can be determined with higher accuracy.

[0261] (Functional example 2 of motion information estimation unit)

[0262] Fig.33 : is a block diagram showing a functional example 2 of the motion information estimation unit 300d. Fig.33 The motion information estimation unit 300d in the embodiment includes a motion direction estimation unit 300d2 and a motion speed estimation unit 300d3. The motion direction estimation unit 300d2 can extract, for example, a feature point with reduced brightness and a feature point with increased brightness, and estimate the motion direction of the eyeball EB based on the positional relationship between the two feature points, or estimate the motion direction of the eyeball EB based on, for example, the direction of the acute angle formed by two straight lines intersecting at the feature point (intersection point) of the corneal reflection image PL4i or the event polarity (positive or negative), as shown in FIG. Fig.32 The movement speed estimation unit 300d3 can extract, for example, a feature point with reduced brightness and a feature point with increased brightness, and estimate the movement speed of the eyeball EB based on the positional relationship between the two feature points and the frame rate of the EVS.

[0263] exist Fig.33 In the functional example 2 of the movement information estimation unit 300d in FIG. 1 , saccade estimation is performed to estimate the target position (arrival position) of the eyeball EB based on the movement direction and movement speed of the eyeball EB. Fig.34 A and Fig.34 B is an explanatory diagram showing the principle of functional example 2 of the motion information estimation unit 300 d . Fig.34 A and Fig.34 B shows the change in eye position and the change in eye velocity over time relative to the same eye movement. Fig.34 A and Fig.34 B shows that changes in eye position and changes in eye velocity over time are correlated with each other, and the eye arrival time can be detected by detecting the maximum velocity.

[0264] (Functional Example 3 of Motion Information Estimation Unit)

[0265] Fig.35 : is a block diagram showing a functional example 3 of the motion information estimation unit 300d. Fig.35The motion information estimation unit 300d in the image processing unit estimates the positions of the feature points corresponding to the intersection points and not detected by the cornea reflection image detection unit 300a (not appearing in the EVS image) from the feature points corresponding to the intersection points and detected by the cornea reflection image detection unit 300a (already appearing in the EVS image), and then extracts the motion direction of the eyeball EB from the estimation result. In this case, the motion information estimation unit 300d includes an undetected intersection estimation unit 300d4 and a motion direction extraction unit 300d5. For example, as Fig.36 As shown, the undetected intersection estimation unit 300d4 estimates the positions of feature points (e.g., feature points of the cornea reflection image PL4Ai) corresponding to the intersection points of the reference pattern BP and not detected by the cornea reflection image detection unit 300a (not appearing in the EVS image) based on the positions of feature points (e.g., feature points of the cornea reflection image PL4Bi) corresponding to the intersection points of the reference pattern BP and detected by the cornea reflection image detection unit 300a (e.g., the positions of the feature points can be estimated Fig.36 The dashed line portion), then the intersection estimation unit 300d4 outputs the estimation result. This can increase the number of feature points, improve the detection accuracy of the sight line detection process 1, and also support the sight line detection process 2.

[0266] (Functional example of pupil detection unit)

[0267] Fig.37 : is a block diagram showing a functional example of the pupil detection unit 300 b. Fig.37 The pupil detection unit 300b in the embodiment includes a motion information acquisition unit 300b1 and a pupil pattern estimation unit 300b2. The motion information acquisition unit 300b1 acquires motion information about the eyeball EB (e.g., at least one of the motion amount, motion direction, and motion speed) as the estimation result of the motion information estimation unit 300d. The pupil pattern estimation unit 300b2 estimates the pattern of the pupil based on the motion information about the eyeball EB acquired by the motion information acquisition unit 300b1, and detects the pupil by pattern detection or the like.

[0268] According to the sight line detection device 40, the sight line detection system 300 includes a motion information estimation unit 300d, thereby further improving the accuracy of sight line detection.

[0269] <13. Visual Line Detection Device According to Modification Examples 6 and 7 of Second Embodiment of the Present Technology>

[0270] Fig.38 A and Fig.38 B shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 6 and 7 of the second embodiment of the present technology.

[0271] like Fig.38 As shown in A, in the visual line detection device according to modification example 6, the patterned light emitted to the eyeball EB includes a first pattern portion PL6A and a second pattern portion PL6B which are rectangular in the same shape (including similar figures) and oriented in different directions.

[0272] like Fig.38 As shown in B, in the visual line detection device according to Modification Example 7, the patterned light emitted to the eyeball EB includes a rectangular first pattern portion PL6 and a triangular second pattern portion PL1'.

[0273] <14. Visual Line Detection Devices According to Modification Examples 4 to 8 of First Embodiment of the Present Technology>

[0274] Fig.39 A to Fig.39 E shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 4 to 8 of the first embodiment of the present technology. Fig.39 A to Fig.39 In the example of E, all reference patterns BP corresponding to the patterned light extend in the radial direction (for example, in a Y shape).

[0275] exist Fig.39 In Modification Example 4 in A, the patterned light PL2 completely matches the reference pattern BP.

[0276] exist Fig.39 In Modification Example 5 in B, patterned light PL2 is shaped to extend along the reference pattern BP so that two of the three straight lines intersect each other and one straight line is slightly spaced apart from the intersection of the two straight lines (eg, 10 pixels or less).

[0277] exist Fig.39 In Modification Example 6 in C, the patterned light PL2 is shaped to extend along the reference pattern BP so that a portion corresponding to an intersection of the reference pattern BP is replaced by a figure (eg, a circle).

[0278] exist Fig.39 In Modification Example 7 in D, the patterned light PL2 is shaped to extend along the reference pattern BP so that a portion corresponding to an intersection of the reference pattern BP is removed.

[0279] exist Fig.39 In Modification Example 8 in E, the patterned light PL2 is shaped to extend along the reference pattern BP so that a portion corresponding to an intersection of the reference pattern BP is removed, and a figure (for example, a circle) is placed in the removed portion.

[0280] <15. Visual Line Detection Device According to Modification Example 9 of First Embodiment of the Present Technology>

[0281] Fig.40FIG. 9 shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Example 9 of the first embodiment of the present technology. Fig.40 In Modification Example 9 in FIG. 1 , the patterned light PL7 has three non-parallel straight lines that intersect at a point. Four line segments extend in the radial direction from the intersection of the three straight lines.

[0282] <16. Visual Line Detection Devices According to Modification Examples 10 to 12 of First Embodiment of the Present Technology>

[0283] Fig.41 A to Fig.41 C shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 10 to 12 of the first embodiment of the present technology.

[0284] exist Fig.41 In Modification Example 10 in A, the patterned light PL9 is in an A shape having three intersection points.

[0285] exist Fig.41 In Modification Example 11 in B, the patterned light PL10 is in an A-shape having horizontal lines extending in the transverse direction and having three intersection points.

[0286] exist Fig.41 In Modification Example 12 in C, the patterned light PL11 is in an A-shape having a horizontal line extending laterally and two oblique lines extending upward, and having three intersection points.

[0287] <17. Visual Line Detection Devices According to Modification Examples 13 to 15 of First Embodiment of the Present Technology>

[0288] Fig.42 A to Fig.42 C shows an eyeball illuminated with patterned light from a visual line detection device according to Modification Examples 13 to 15 of the first embodiment of the present technology.

[0289] exist Fig.42 In Modification Example 13 in A, the patterned light PL12 is shaped so that a curve and two straight lines intersect each other at a certain point and extend radially from the intersection point. The patterned light PL12 may be shaped so that a curve and two straight lines intersect each other at a plurality of points.

[0290] exist Fig.42 In Modification Example 14 in B, the patterned light PL13 is shaped so that two curved lines and a straight line intersect each other at a certain point. The patterned light PL13 may be shaped so that two curved lines and a straight line intersect each other at a plurality of points.

[0291] exist Fig.42In Modification Example 15 in C, the patterned light PL14 is shaped so that three curved lines intersect at a certain point. The patterned light PL14 may be shaped so that three curved lines intersect at a plurality of points.

[0292] <18. Variation 1 of Patterned Light>

[0293] Fig.43 Variation 1 of patterned light applicable to the visual line detection device according to the present technology and a corneal reflection image of the patterned light are shown. Fig.43 2 shows patterned light that is shaped like a star in addition to being shaped into polygons such as triangles, pentagons, and hexagons. Fig.43 , EVS images in each direction of eye movement (eg, horizontal, vertical, and diagonal) are shown below each type of patterned light. Fig.43 It is shown that at least one characteristic point (intersection point) of the corneal reflection image can be detected from patterned light of any shape with respect to horizontal movement, vertical movement, and diagonal movement of the eyeball.

[0294] <19. Patterned Light Variation 2>

[0295] Fig.44 Variation 2 of patterned light applicable to a visual line detection device according to the present technology and a corneal reflection image of the patterned light are shown. Fig.44 2 shows patterned light that is shaped like a star-shaped frame in addition to being shaped like a polygonal frame such as a triangle frame, a pentagonal frame, and a hexagonal frame. Fig.44 , EVS images in each direction of eye movement (eg, horizontal, vertical, and diagonal) are shown below each type of patterned light. Fig.44 It is shown that at least one characteristic point (intersection point) of the corneal reflection image can be detected from patterned light of any shape with respect to horizontal movement, vertical movement, and diagonal movement of the eyeball.

[0296] <20. Other Modification Examples of the Present Technology>

[0297] For example, the number of light sources and the number of optical elements can be appropriately changed. The number of light sources and the number of optical elements can be determined according to the number of divisions of the patterned light (the number of pattern portions). Specifically, when the light source emits a pattern portion of patterned light, the number of light sources can be set to be equal to the number of pattern portions. When the light emitted by the light source is formed into a pattern portion by an optical element, the number of optical elements can be set to be equal to the number of pattern portions, and the number of light sources can be set to be equal to or less than the number of pattern portions.

[0298] For example, the imaging element 200 a of the imaging system is not limited to the EVS and may be a general image sensor.

[0299] In the visual line detection devices according to the above-described embodiments and modified examples, the types, numbers, shapes, and arrangements of light sources and optical elements may be appropriately changed.

[0300] For example, at least a part of the visual line detection devices according to the embodiment and the modified example may be combined without inconsistency.

[0301] This technology can also be configured as follows:

[0302] (1) A sight line detection device comprising:

[0303] a patterned light irradiation system, the patterned light irradiation system irradiating the eyeball with patterned light;

[0304] an imaging system that captures an image of the eye illuminated with the patterned light; and

[0305] A sight line detection system, the sight line detection system detecting a sight line as the direction of the eyeball based on the captured image of the imaging system, wherein

[0306] The patterned light has a shape along a reference pattern, the reference pattern has at least three lines that are not parallel to each other, the reference pattern has an intersection, and at least two lines of the at least three lines cross each other at the intersection.

[0307] (2) The visual line detection device according to (1), wherein each of the at least three lines intersects with at least one of the other lines.

[0308] (3) The visual line detection device according to (1) or (2), wherein each of the at least three lines intersects at least two of the other lines.

[0309] (4) The visual line detection device according to any one of (1) to (3), wherein the reference pattern has at least three intersection points that are not located on the same straight line.

[0310] (5) The visual line detection device according to any one of (1) to (4), wherein the at least three lines intersect each other at the same intersection point.

[0311] (6) The visual line detection device according to any one of (1) to (5), wherein the at least three lines form continuous acute angles around the intersection point.

[0312] (7) The visual line detection device according to (5) or (6), wherein the at least three lines extend radially from the intersection.

[0313] (8) The visual line detection device according to any one of (1) to (7), wherein the reference pattern has a plurality of patterns, each of the plurality of patterns including at least two lines intersecting each other.

[0314] (9) The visual line detection device according to (8), wherein the plurality of patterns include at least two patterns having the same shape.

[0315] (10) The visual line detection device according to (9), wherein the orientations of the plurality of patterns are in the same direction.

[0316] (11) The visual line detection device according to (9), wherein the orientations of the plurality of patterns are in different directions.

[0317] (12) The visual line detection device according to (8), wherein the plurality of patterns include at least two patterns having different shapes.

[0318] (13) A line of sight detection device according to any one of (8) to (12), wherein the patterned light illumination system illuminates the eyeball with a portion of the patterned light, and the portion of the patterned light corresponds to a pattern among the multiple patterns that can detect the intersection.

[0319] (14) A line of sight detection device according to any one of (1) to (13), wherein the line of sight detection system includes a cornea reflection image detection unit, and the cornea reflection image detection unit detects feature points of the cornea reflection image of the patterned light from the captured image.

[0320] (15) The line of sight detection device according to (14), wherein the cornea reflection image detection unit associates the intersection point with the feature point based on information indicating an angle and / or direction near the feature point of the cornea reflection image in the captured image.

[0321] (16) The visual line detection device according to (14) or (15), wherein the cornea reflection image detection unit extracts pixels whose brightness has increased in the captured image as pixels constituting the feature points.

[0322] (17) A gaze detection device according to any one of (14) to (16), wherein the gaze detection system includes a motion information estimation unit, which estimates at least one of the movement amount, movement direction and movement speed of the eyeball based on the detection result in the corneal reflection image detection unit.

[0323] (18) A line of sight detection device according to (17), wherein the motion information estimation unit estimates the position of a feature point corresponding to the intersection and not detected by the cornea reflection image detection unit based on the feature point corresponding to the intersection and detected by the cornea reflection image detection unit, and then extracts the movement direction of the eyeball from the estimation result.

[0324] (19) A line of sight detection device according to any one of (14) to (18), wherein the line of sight detection system detects the line of sight based on at least a captured image of the imaging system and a detection result in the corneal reflection image detection unit.

[0325] (20) The visual line detection device according to any one of (1) to (19), wherein the patterned light irradiation system includes a light source that emits the patterned light.

[0326] (21) A line of sight detection device according to any one of (1) to (19), wherein the patterned light irradiation system comprises: a light source; and an optical element that shapes the light from the light source to generate the patterned light.

[0327] (22) The visual line detection device according to any one of (1) to (21), wherein the imaging system includes an event sensor.

[0328] [reference numerals list]

[0329] 10, 20, 30, 40 Line of sight detection device 100 Patterned light irradiation system

[0330] 100a light source

[0331] 100a1 First light source (light source)

[0332] 100a2 Second light source (light source)

[0333] 100a3 The third light source (light source)

[0334] 200 Imaging System

[0335] 300 sight detection system

[0336] 300a Corneal reflection image detection unit

[0337] 300d Motion Information Estimation Unit

Claims

1. A sight line detection device, comprising: a patterned light irradiation system, the patterned light irradiation system irradiating the eyeball with patterned light; an imaging system that captures an image of the eye illuminated with the patterned light; as well as A sight line detection system, the sight line detection system detecting a sight line as the direction of the eyeball based on the captured image of the imaging system, wherein The patterned light has a shape along a reference pattern, the reference pattern has at least three lines that are not parallel to each other, the reference pattern has an intersection, and at least two lines of the at least three lines cross each other at the intersection.

2. The sight line detection device according to claim 1, wherein: Each of the at least three lines crosses at least one of the other lines.

3. The sight line detection device according to claim 1, wherein: Each of the at least three lines crosses at least two of the other lines.

4. The sight line detection device according to claim 1, wherein: The reference pattern has at least three intersection points that are not located on the same straight line.

5. The sight line detection device according to claim 1, wherein: The at least three lines intersect each other at the same intersection point.

6. The sight line detection device according to claim 5, wherein: The at least three lines form continuous acute angles around the intersection point.

7. The sight line detection device according to claim 5, wherein: The at least three lines extend radially from the intersection point.

8. The sight line detection device according to claim 1, wherein: The reference pattern has a plurality of patterns, each of which includes at least two lines crossing each other.

9. The sight line detection device according to claim 8, wherein: The plurality of patterns include at least two patterns having the same shape.

10. The sight line detection device according to claim 8, wherein: The plurality of patterns include at least two patterns having different shapes.

11. The sight line detection device according to claim 8, wherein: The patterned light irradiation system irradiates the eyeball with a portion of the patterned light, the portion of the patterned light corresponding to a pattern in the plurality of patterns in which the intersection can be detected.

12. The sight line detection device according to claim 1, wherein: The sight line detection system includes a cornea reflection image detection unit that detects feature points of a cornea reflection image of the patterned light from the captured image.

13. The sight line detection device according to claim 12, wherein: The cornea reflection image detection unit associates the intersection point with the feature point based on information indicating an angle and / or a direction near a feature point of the cornea reflection image in the captured image.

14. The sight line detection device according to claim 12, wherein: The cornea reflection image detection unit extracts pixels whose brightness has increased in the captured image as pixels constituting the feature points.

15. The sight line detection device according to claim 12, wherein: The sight line detection system includes a motion information estimation unit, which estimates at least one of the motion amount, motion direction and motion speed of the eyeball based on the detection result in the cornea reflection image detection unit.

16. The sight line detection device according to claim 15, wherein: The motion information estimation unit estimates the positions of feature points corresponding to the intersection and not detected by the cornea reflection image detection unit based on the feature points corresponding to the intersection and detected by the cornea reflection image detection unit, and then extracts the movement direction of the eyeball from the estimation result.

17. The sight line detection device according to claim 12, wherein: The line of sight detection system detects the line of sight based on at least a captured image of the imaging system and a detection result in the cornea reflection image detection unit.

18. The sight line detection device according to claim 1, wherein: The patterned light illumination system includes a light source that emits the patterned light.

19. The sight line detection device according to claim 1, wherein: The patterned light irradiation system comprises: Light source; and An optical element shapes the light from the light source to generate the patterned light.

20. The sight line detection device according to claim 1, wherein: The imaging system includes an event sensor.

Images