Corneal reflection image specific device and line-of-sight detection device

Abstract

To provide a corneal reflected image identifying device and a gaze detection device, which can detect a gaze with high precision even when a person to be detected is wearing an eye covering object.SOLUTION: A corneal reflected image identifying device includes: a light projecting part 110 for projecting light, toward an eye, from two light sources 110a and 110b which are spaced apart from each other; a camera 120 for capturing an image of the eye on which light is projected; a distance calculation unit 152d configured to use the captured image to calculate a distance between two reflected images for each combination of two reflected images among a plurality of reflected images obtained by light projected from the light sources 110a and 110b being reflected in the vicinity of the eye; and a corneal reflected image identification unit 152g configured to exclude a reflected image of light projected from the light sources 110a and 110b being reflected by an eye covering object based on predetermined features and identify a corneal reflected image to be used for gaze detection, wherein the predetermined features include the distance calculated by the distance calculation unit 152d.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a corneal reflection image identification device and a gaze detection device.

Background Art

[0002] Conventionally, there is a technique of using a plurality of synchronized cameras to generate a plurality of corneal reflection images and selecting the correct corneal reflection image for each camera by using the distance and angle of the pair (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When light is projected onto the eye and the gaze is detected using the reflection image of the light on the cornea, if the subject is wearing glasses, the image reflected by the lens, frame, etc. of the glasses may be misdetected as the reflected light of the cornea. And when such misdetection occurs, there is a problem that the detection accuracy of the gaze decreases.

[0005] The technique described in the above patent document generates a plurality of corneal reflection images by using a plurality of synchronized cameras. However, the corneal reflection light corresponding to one camera is one, and by selecting the correct corneal reflection image for each camera, the gaze detection range is expanded. Therefore, the technique described in the above patent document is a technique premised on the existence of a corneal reflection image reflected by the cornea. When detecting the gaze of a glasses wearer, there still exists a problem that the image reflected by the lens, frame, etc. of the glasses is misdetected as the reflected light of the cornea.

[0006] Therefore, an object of the present invention is to provide a corneal reflection image identification device and a gaze detection device capable of accurately detecting a gaze even when a subject is wearing an object that covers the eyes.

Means for Solving the Problems

[0007] According to one embodiment, a corneal reflection image identification device is provided. This corneal reflection image identification device includes a light projecting unit that projects light toward the eyes from two light sources spaced apart from each other, an imaging unit that images the illuminated eyes, a distance calculation unit that calculates the distance between two reflection images for each combination of two reflection images out of a plurality of reflection images obtained by reflection of the light projected from the light sources in the vicinity of the eyes using the captured image, and a corneal reflection image identification unit that identifies a corneal reflection image used for gaze detection, excluding the reflection image reflected by an object covering the eyes from the light projected from the light sources, based on a predetermined characteristic value. The predetermined characteristic value includes the distance calculated by the distance calculation unit.

[0008] In this corneal reflection image identification device, it is preferable that the corneal reflection image identification unit identifies, as the corneal reflection image used for gaze detection, one of the reflection images included in a combination of two reflection images having a distance equal to or less than a first threshold value, excluding combinations of two reflection images having a distance exceeding the first threshold value.

[0009] In this corneal reflection image identification device, an angle calculation unit is provided that calculates, for each combination of two reflection images, the angle between the straight line connecting the two reflection images with respect to the straight line connecting the two light sources when viewed from the imaging direction of the imaging unit. The predetermined characteristic value includes the angle calculated by the angle calculation unit. It is preferable that the corneal reflection image identification unit identifies, as the corneal reflection image used for gaze detection, one of the reflection images included in a combination of two reflection images having an angle equal to or less than a second threshold value, excluding combinations of two reflection images having an angle exceeding the second threshold value.

[0010] In this corneal reflection image identification device, an area calculation unit that calculates the area of the reflection image is provided for each combination of two reflection images. The predetermined characteristic value includes the area calculated by the angle calculation unit. The corneal reflection image identification unit preferably identifies, as the corneal reflection image used for line-of-sight detection, one reflection image included in the combination of two reflection images whose area is equal to or less than a third threshold, excluding the combination of two reflection images whose area exceeds the third threshold.

[0011] In this corneal reflection image identification device, it is preferable to include a threshold correction unit that corrects the first threshold to a larger value as the distance from the light source to the face increases.

[0012] In this corneal reflection image identification device, it is preferable to include a threshold correction unit that corrects the third threshold to a larger value as the distance from the light source to the face increases.

[0013] According to one embodiment, a line-of-sight detection device is provided. This line-of-sight detection device includes a line-of-sight detection unit that performs line-of-sight detection by the corneal reflection method based on the corneal reflection image identified by the above corneal reflection image identification device.

Advantages of the Invention

[0014] According to the present invention, even when the subject is wearing an object that covers the eyes, it is possible to provide a corneal reflection image identification device and a line-of-sight detection device capable of detecting the line of sight with high accuracy.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0016] FIG. 1 is a schematic configuration diagram of an occupant support system 100 in which a corneal reflex image identification device and a line-of-sight detection device are mounted according to one embodiment. The occupant support system 100 is mounted on a vehicle such as an automobile and detects the line of sight of an occupant such as a driver. In the following description, the case where the occupant is a driver will be described as an example. The occupant support system 100 includes a light projecting unit 110 that projects light toward the driver's face, a camera 120 that photographs the driver's face and generates an image, a display device 130 that displays warnings or the like for the driver, a speaker 140 that emits warnings or the like for the driver by voice, and a control device (ECU) 150. Each of the light projecting unit 110, the camera 120, the display device 130, the speaker 140, and the control device 150 is communicably connected via an in-vehicle network conforming to a standard such as a Controller Area Network (CAN).

[0017] The light projection unit 110 has two light sources spaced apart from each other, and projects light from the two light sources toward the driver's eyes. The two light sources are composed of, for example, near-infrared light sources (near-infrared LEDs) that irradiate near-infrared light.

[0018] The camera 120 is an aspect of the imaging unit, and includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light or near-infrared light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of the area to be imaged on the two-dimensional detector. The camera 120 is provided toward the assumed position of the driver near the dashboard, steering column, or windshield inside the vehicle, and captures an image of the driver's face. The camera 120 captures the driver's eyes irradiated with light by the light projection unit 110 every predetermined imaging cycle (for example, 1 / 30 second to 1 / 10 second), and generates an image representing the driver's eyes. Note that the image obtained by the camera 120 is preferably a color image. Further, the camera 120 may be composed of a stereo camera, and may be configured to acquire the distance to each structure on the image from the parallax of the left and right images. Each time the camera 120 generates an image, the generated image is output to the control device 150 via the in-vehicle network. Note that, as will be described later, the camera 120 may be integrally configured with the light projection unit 110.

[0019] The display device 130 is composed of, for example, a liquid crystal display device (LCD), etc., is provided near the meter panel or the dashboard, etc., and displays a warning to the driver as necessary. The speaker 140 is provided near the dashboard, etc., and issues an audible warning to the driver as necessary.

[0020] The control device 150 is a component that controls the entire occupant support system 100. The corneal reflection image identification device and the line-of-sight detection device according to the present embodiment are configured to include a control device 150, a light projection unit 110, and a camera 120. The control device 150 includes a processor 152, a memory 154, and a communication interface (I / F) 156. The processor 152 includes one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 152 may further include other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The memory 154 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory, and stores programs, data, etc. related to the processing according to the present embodiment. The communication interface 156 includes an interface circuit for connecting the control device 150 to an in-vehicle network.

[0021] The control device 150 detects the driver's line of sight. In the present embodiment, detecting the line of sight means detecting the direction (angle of the line of sight) of the line of sight. As a method for calculating the line-of-sight angle, a method of calculating the line-of-sight angle from the distance between the pupil center and the Purkinje image using the reflection image (Purkinje image) of a near-infrared light source on the corneal surface, a method of using the corneal curvature center obtained from the Purkinje image and calculating the straight line connecting the corneal curvature center and the pupil center as the line-of-sight angle, etc. are known. In the present embodiment, the driver's line of sight is detected using these methods using corneal reflection images (hereinafter, also referred to as the corneal reflection method).

[0022] When detecting the driver's line of sight, if the driver is wearing glasses, the near-infrared light projected by the light projection unit 110 may be reflected by the frame or lens of the glasses, etc., and the resulting glasses reflection image may be misdetected as the reflected light on the pupil or the cornea. When such a misdetection occurs, the line-of-sight detection accuracy decreases. In particular, in the case of a monocular camera, since interpolation by increasing the number of cameras or countermeasures by geometric methods cannot be performed, it is difficult to appropriately handle misdetection due to the reflection image reflected by the glasses, and the line-of-sight detection accuracy is likely to decrease.

[0023] In this embodiment, by projecting light from the light projecting unit 110 having two light sources separated from each other toward the driver's eyes, two corneal reflection images are caused to appear for one eye. Then, based on the distance between the two points of the reflection image, the angle of the line connecting the two points of the reflection image, and the area of the reflection image, the spectacle reflection image is excluded, and it is suppressed that the spectacle reflection image is erroneously detected as a corneal reflection image. Thereby, even in the case of a spectacle wearer, more accurate line-of-sight detection becomes possible. In this embodiment, spectacles are exemplified as an object covering the eyes, but similarly, even when other transparent or translucent objects covering the eyes such as sunglasses, face shields, and goggles are worn, false detection due to the reflection image reflected by these objects is suppressed, and more accurate line-of-sight detection is realized.

[0024] FIGS. 2 and 3 are schematic views showing a camera unit 160 in which the light projecting unit 110 and the camera 120 are integrated. FIGS. 2 and 3 show the appearance of the camera unit 160 as viewed from the optical axis direction of the imaging optical system of the camera 120. The camera unit 160 shown in FIG. 2 includes the camera 120, a light source 110a provided on the left side of the camera 120 in the horizontal direction when the camera unit 160 is viewed from the front, and a light source 110b provided on the right side of the camera 120 in the horizontal direction when the camera unit 160 is viewed from the front. The camera unit 160 shown in FIG. 3 includes the camera 120, a light source 110a provided so as to surround the periphery of the camera 120, and a light source 110b provided on the right side of the camera 120 in the horizontal direction when the camera unit 160 is viewed from the front. The light sources 110a and 110b shown in FIGS. 2 and 3 are each composed of an aggregate of a plurality of point light sources. Hereinafter, the case where the light projecting unit 110 and the camera 120 are composed of the camera unit 160 shown in FIG. 2 or FIG. 3 will be described as an example.

[0025] FIG. 4 is a diagram showing an image 50 representing the driver's eye 10, which is generated by photographing the eye of a driver wearing glasses with the camera unit 160 shown in FIG. 2 or FIG. 3. As shown in FIG. 4, the contour 12 of the eye 10 and the pupil 20 are represented in the image 50, and near the pupil 20, a corneal reflection image 30a of the light of the light source 110a reflected by the cornea and a corneal reflection image 30b of the light of the light source 110b reflected by the cornea are represented. Also, near the pupil 20, two spectacle reflection images 40 and 42 reflected by the glasses are represented. Specifically, the spectacle reflection images 40 and 42 are reflection images of the light of the light sources 110a and 110b reflected by the frame or lens of the glasses or the like.

[0026] As shown in FIG. 4, the spectacle reflection images 40 and 42 are represented in the image 50 generated by photographing the eye of a driver wearing glasses. If it is erroneously determined that the spectacle reflection images 40 and 42 are reflection images on the cornea and the driver's line of sight is detected using the spectacle reflection image 40 or the spectacle reflection image 42, the driver's line of sight will be erroneously detected.

[0027] Since the corneal reflection images 30a and 30b are reflection images of the light sources 110a and 110b on the cornea, the distance between the corneal reflection image 30a and the corneal reflection image 30b is a value that is pre-determined to fall within a certain range according to the distance between the light sources 110a and 110b.

[0028] On the other hand, since the spectacle reflection image 40 or the spectacle reflection image 42 is an image of the light of the light source 110a or 110b reflected by the glasses outside the cornea, the distance between the spectacle reflection image 40 and the spectacle reflection image 42, or the distance between either one of the spectacle reflection images 40 and 42 and either one of the corneal reflection images 30a and 30b, is usually larger than the distance between the corneal reflection image 30a and the corneal reflection image 30b. Also, the distance between the spectacle reflection image 40 and the spectacle reflection image 42 is usually larger than the distance between the corneal reflection image 30a and the corneal reflection image 30b.

[0029] In this embodiment, two reflected images are extracted from the image 50, the distance between the two extracted reflected images is obtained, and if the obtained distance is equal to or less than a predetermined value, it is determined that the two extracted reflected images are the corneal reflection image 30a and the corneal reflection image 30b. In this case, the direction of the driver's line of sight is accurately detected by the corneal reflection method using the corneal reflection image 30a or the corneal reflection image 30b. On the other hand, two reflected images are extracted from the image 50, the distance between the two extracted reflected images is obtained, and if the obtained distance is greater than the predetermined value, it is determined that at least one of the two extracted reflected images is the spectacle reflection image 40 or the spectacle reflection image 42. In this case, by not performing the line-of-sight detection based on these two reflected images, the misjudgment of the line of sight is suppressed.

[0030] On the other hand, it may rarely occur that the distance between either one of the spectacle reflection image 40 and the spectacle reflection image 42 and either one of the corneal reflection image 30a and the corneal reflection image 30b becomes equal to or less than the above-described predetermined value. Also, it may rarely occur that the distance between the spectacle reflection image 40 and the spectacle reflection image 42 becomes equal to or less than the above-described predetermined value. For this reason, the spectacle reflection image is excluded based on the angle of the line connecting two points of the reflected image.

[0031] As long as the reflecting surface does not have a complex shape, the direction connecting the two light sources 110a and 110b and the direction connecting the reflected images on the reflecting surface are the same. Therefore, the angle of the straight line connecting the two corneal reflection images depends on the angle of the straight line connecting the two light sources 110a and 110b, and substantially coincides with the angle of the straight line connecting the two light sources 110a and 110b when viewed from the imaging direction. As shown in FIGS. 2 and 3, in the camera unit 160, since the two light sources 110a and the light source 110b are arranged in the horizontal direction, as shown in FIG. 4, the corneal reflection image 30a obtained by reflecting the light of the light source 110a on the cornea and the corneal reflection image 30b obtained by reflecting the light of the light source 110b on the cornea are also represented in a state of being arranged horizontally in the image 50.

[0032] On the one hand, when the reflected image includes a spectacle reflection image, the spectacle reflection image is an image formed by the light from the light source 110a or the light source 110b reflected by the reflecting surface of the spectacle such as the frame or lens of the spectacle. Since the reflecting surface of the spectacle faces in a random direction, when viewed from the photographing direction, the angle of the straight line connecting the spectacle reflection image and the corneal reflection image does not coincide with the angle of the straight line connecting the two light sources 110a and 110b. Similarly, the angle of the straight line connecting the two spectacle reflection images does not coincide with the angle of the straight line connecting the two light sources 110a and 110b.

[0033] Therefore, in FIG. 4, the spectacle reflection image 40 and the spectacle reflection image 42 basically do not line up horizontally, and the line connecting the spectacle reflection image 40 and the spectacle reflection image 42 is not horizontal. Similarly, the line connecting either one of the spectacle reflection image 40 and the spectacle reflection image 42 and either one of the corneal reflection image 30a and the corneal reflection image 30b is not horizontal.

[0034] In this embodiment, two reflected images are extracted from the image 50, the angle of the line connecting the two extracted reflected images is obtained, and if the obtained angle is horizontal, it is determined that the two extracted reflected images are the corneal reflection image 30a and the corneal reflection image 30b. In this case, the direction of the driver's line of sight is accurately detected by the corneal reflection method using the corneal reflection image 30a or the corneal reflection image 30b. On the other hand, two reflected images are extracted from the image 50, the angle of the line connecting the two extracted reflected images is obtained, and if the obtained angle is not horizontal, it is determined that at least one of the two extracted reflected images is the spectacle reflection image 40 or the spectacle reflection image 42. In this case, by not performing the line-of-sight detection based on these two reflected images, the false determination of the line of sight is suppressed.

[0035] Furthermore, the area of the corneal reflection image is smaller than the area of the eyeglass reflection image due to the curvature and size of the reflection surface. The sizes of the light sources 110a and 110b are preset so that the corneal reflection images 30a and 30b are of the minimum size required for detecting the line of sight by the corneal reflection method. Therefore, the size of the corneal reflection image 30a formed by the light of the light source 110a reflected by the cornea in the image 50 is in accordance with the size of the light source 110a. Similarly, the size of the corneal reflection image 30b formed by the light of the light source 110b reflected by the cornea in the image 50 is in accordance with the size of the light source 110b. On the other hand, the eyeglass reflection image 40 or the eyeglass reflection image 42 is an image formed by the light of the light source 110a or the light source 110b reflected by the reflection surface of the eyeglass such as the frame or lens of the eyeglass. Due to the difference in curvature between the reflection surface of the eyeglass and the cornea or the difference in size between the reflection surface of the eyeglass and the cornea, the size of the eyeglass reflection image 40 or the eyeglass reflection image 42 is larger than that of the corneal reflection image 30a or the corneal reflection image 30b.

[0036] In this embodiment, a reflection image is extracted from the image 50, the area of the extracted reflection image is obtained, and if the obtained area is equal to or less than a predetermined value, it is determined that the extracted reflection image is the corneal reflection image 30a or the corneal reflection image 30b. In this case, the direction of the driver's line of sight is accurately detected by the corneal reflection method using the corneal reflection image 30a or the corneal reflection image 30b. On the other hand, a reflection image is extracted from the image 50, the area of the extracted reflection image is obtained, and if the obtained area is larger than the predetermined value, it is determined that the extracted reflection image is the eyeglass reflection image 40 or the eyeglass reflection image 42. In this case, by not performing line-of-sight detection based on the extracted reflection image, false determination of the line of sight is suppressed.

[0037] Note that when there is clearly no eyeglass reflection image in the image, the process of identifying the corneal reflection image based on the distance between two points of the reflection image, the angle of the line connecting two points of the reflection image, or the area of the reflection image does not need to be performed. For example, when only two reflection images exist in the image, there is no eyeglass reflection image, and the two reflection images are considered to be corneal reflection images formed by the light of the two light sources 110a and 110b reflected by the cornea, so the process of identifying the corneal reflection image does not need to be performed.

[0038] FIG. 5 is a schematic diagram showing functional blocks of the processor 152 of the control device 150 for performing the above-described processing. The processor 152 of the control device 150 includes a feature extraction unit 152a, a reflection image extraction unit 152b, a candidate selection unit 152c, a distance calculation unit 152d, an angle calculation unit 152e, an area calculation unit 152f, a corneal reflection image identification unit 152g, a threshold correction unit 152h, and a gaze detection unit 152i. Each of these units included in the processor 152 is a functional module realized by, for example, a computer program operating on the processor 152. That is, the functional blocks of the processor 152 are composed of the processor 152 and a program (software) for operating the same. Further, the program may be recorded in the memory 154 included in the control device 150 or a recording medium connected from the outside. Alternatively, each of these units included in the processor 152 may be a dedicated arithmetic circuit provided in the processor 152.

[0039] The feature extraction unit 152a of the processor 152 determines whether or not the face of the driver is represented in the image generated by the camera 120 capturing the driver. When the face of the driver is represented, the feature extraction unit 152a extracts an image corresponding to the face of the driver. Further, when the face of the driver is represented in the image, the feature extraction unit 152a determines whether or not eyes (pupils) are represented in the extracted image corresponding to the face of the driver. At this time, the feature extraction unit 152a extracts an image of the face of the driver from the image generated by the camera 120, for example, by template matching between a template image representing a face and the image generated by the camera 120, or by inputting the image generated by the camera 120 to a discriminator learned for face detection. Similarly, the feature extraction unit 152a extracts the eye contour 12 as shown in FIG. 4 and further extracts the pupil 20, for example, by template matching between a template image representing eyes and the image generated by the camera 120, or by inputting the image generated by the camera 120 to a discriminator learned for eye detection.

[0040] When the reflection image extraction unit 152b of the processor 152 detects that the eyes of the driver are represented in the image generated by the camera 120, it extracts a plurality of reflection images obtained by the reflection of the near-infrared light projected from the light sources 110a and 110b in the vicinity of the eyes. More specifically, the reflection image extraction unit 152b extracts the reflection images of the near-infrared light in the vicinity of the pupil 20, thereby extracting the reflection images of the near-infrared light projected from the light source 110a and the light source 110b. The reflection images of the near-infrared light extracted by the reflection image extraction unit 152b include the corneal reflection images 30a and 30b, as well as the spectacle reflection images 40 and 42.

[0041] The candidate selection unit 152c of the processor 152 selects a combination of two reflection images that are candidates for the corneal reflection image from among the plurality of reflection images of the near-infrared light extracted by the reflection image extraction unit 152b. When there are reflection images that have already been determined not to be corneal reflection images through the processing described later among the plurality of reflection images of the near-infrared light extracted by the reflection image extraction unit 152b, the candidate selection unit 152c excludes the reflection images determined not to be corneal reflection images and selects a combination of two reflection images that are candidates for the corneal reflection image.

[0042] The distance calculation unit 152d of the processor 152 calculates the distance between the two reflection images selected by the candidate selection unit 152c for each combination of the two reflection images. At this time, the distance calculation unit 152d may calculate the distance between the two reflection images by obtaining the centroid of each reflection image and calculating the distance between the centroids. Alternatively, the distance calculation unit 152d may calculate the distance between the two reflection images by obtaining the number of pixels between these centroids on the image.

[0043] FIG. 6 is a schematic diagram showing how the distance calculation unit 152d calculates the distance between two candidates for the reflected image in the example of FIG. 4. As shown in FIG. 6, when the two reflected images selected by the candidate selection unit 152c are the corneal reflection image 30a and the corneal reflection image 30b, the distance calculation unit 152d calculates the distance L1 between the corneal reflection image 30a and the corneal reflection image 30b. Further, when the two reflected images selected by the candidate selection unit 152c are the corneal reflection image 30a and the glasses reflection image 40, the distance calculation unit 152d calculates the distance L2 between the corneal reflection image 30a and the glasses reflection image 40. Further, when the two reflected images selected by the candidate selection unit 152c are the corneal reflection image 30b and the glasses reflection image 42, the distance calculation unit 152d calculates the distance L3 between the corneal reflection image 30b and the glasses reflection image 42.

[0044] The angle calculation unit 152e of the processor 152 calculates, for a combination of two reflected images selected by the candidate selection unit 152c, the angle of the line connecting the two reflected images with respect to the line connecting between the two light sources 110a and 110b when viewed from the imaging direction by the camera 120. As a more specific example, when the two light sources 110a and 110b are arranged in the horizontal direction, the angle calculation unit 152e calculates the angle formed by the line connecting the combination of the two reflected images selected by the candidate selection unit 152c with the horizontal. At this time, the distance calculation unit 152d may obtain the center of gravity of each reflected image and calculate the angle formed by the line connecting the centers of gravity with the horizontal.

[0045] FIG. 7 is a schematic diagram showing a state in which the angle calculation unit 152e calculates the angle formed by the line connecting two candidate reflected images to be horizontal. In the example shown in FIG. 7, two corneal reflection images 30a and corneal reflection image 30b, and one spectacle reflection image 44 are represented in the image 50. Further, FIG. 8 is a diagram showing an enlarged view of the positional relationship among the corneal reflection image 30a, corneal reflection image 30b, and spectacle reflection image 44 shown in FIG. 7. As shown in FIGS. 7 and 8, when the two reflected images selected by the candidate selection unit 152c are the corneal reflection image 30a and the corneal reflection image 30b, the angle θ1 formed by the vector A starting from the corneal reflection image 30b and ending at the corneal reflection image 30a with the horizontal axis L is calculated. Also, when the two reflected images selected by the candidate selection unit 152c are the corneal reflection image 30a and the spectacle reflection image 44, the angle θ2 formed by the vector B starting from the corneal reflection image 30a and ending at the spectacle reflection image 44 with the horizontal axis L is calculated. Note that the horizontal axis L may be corrected according to the driver's face orientation. For example, when the driver's face is tilted when looking at the screen of the in-vehicle navigation device, the tilt of the horizontal axis L may be corrected accordingly.

[0046] The area calculation unit 152f of the processor 152 calculates the area of the two reflected images in the combination selected by the candidate selection unit 152c. The area calculation unit 152f may calculate the area of the reflected image, for example, by obtaining the number of pixels in the area of the reflected image on the image.

[0047] The corneal reflection image identification unit 152g of the processor 152 identifies a combination of corneal reflection images, excluding the spectacle reflection image reflected by the light projected from the light sources 110a and 110b on the spectacle, based on a predetermined characteristic value, and identifies the corneal reflection image to be used for gaze detection. The predetermined characteristic value includes the distance calculated by the distance calculation unit 152d, the angle calculated by the angle calculation unit 152e, or the area calculated by the area calculation unit 152f. The corneal reflection image identification unit 152g identifies a combination of corneal reflection images from among the combinations of two reflected images that are candidates for the corneal reflection image selected by the candidate selection unit 152c based on these characteristic values.

[0048] The specific part 152g of the corneal reflection image preferably identifies a combination of corneal reflection images from among the reflection images extracted by the feature extraction unit 152a based on all of these distances, angles, and areas, and identifies one corneal reflection image included in the identified combination as the corneal reflection image to be used for gaze detection.

[0049] On the other hand, the specific part 152g of the corneal reflection image may identify a combination of corneal reflection images from among the reflection images extracted by the feature extraction unit 152a based on at least one of the distance, angle, and area, and identify one corneal reflection image included in the identified combination as the corneal reflection image to be used for gaze detection. However, from the viewpoint of suppressing the misdetection of the glasses reflection image as the corneal reflection image, the reliability of the method based on the distance is the highest, and the reliability decreases in the order of the method based on the angle and the method based on the area. Therefore, when identifying the corneal reflection image based on one of the distance, angle, and area, it is preferable to identify the corneal reflection image based on the distance. Also, when identifying the corneal reflection image based on two of the distance, angle, and area, it is preferable to identify the corneal reflection image based on the distance and the angle.

[0050] More specifically, the specific part 152g of the corneal reflection image excludes combinations of two reflection images where the distance calculated by the distance calculation unit 152d exceeds the first threshold value, and identifies combinations of two reflection images where the distance is equal to or less than the first threshold value as combinations of corneal reflection images, and identifies one reflection image included in the identified combination as the corneal reflection image to be used for gaze detection. Also, the specific part 152g of the corneal reflection image excludes combinations of two reflection images where the angle calculated by the angle calculation unit 152e exceeds the second threshold value, and may identify combinations of two reflection images where the angle is equal to or less than the second threshold value as combinations of corneal reflection images, and identify one reflection image included in the identified combination as the corneal reflection image to be used for gaze detection. Further, the specific part 152g of the corneal reflection image excludes combinations of two reflection images where the area calculated by the area calculation unit 152f exceeds the third threshold value, and may identify combinations of two reflection images where the area is equal to or less than the third threshold value as combinations of corneal reflection images, and identify one reflection image included in the identified combination as the corneal reflection image to be used for gaze detection.

[0051] Note that by using either one of the corneal reflection images 30a formed by reflection of the light from the light source 110a on the cornea and the corneal reflection image 30b formed by reflection of the light from the light source 110b on the cornea, it is possible to detect the line of sight by the corneal reflection method. By determining in advance which of the corneal reflection images 30a and 30b is to be used for line-of-sight detection, the corneal reflection image specifying unit 152g specifies one corneal reflection image included in the specified combination of corneal reflection images as the corneal reflection image to be used for line-of-sight detection.

[0052] When the threshold correction unit 152h of the processor 152 specifies a corneal reflection image based on the distance or area, it corrects the threshold to be compared with the distance or area based on the distance from the light sources 110a and 110b to the driver's face. The distance between the corneal reflection image 30a formed by reflection of the light from the light source 110a on the cornea and the corneal reflection image 30b formed by reflection of the light from the light source 110b on the cornea changes according to the distance from the light sources 110a and 110b to the driver's face. Similarly, the area of the corneal reflection image 30a formed by reflection of the light from the light source 110a on the cornea or the area of the corneal reflection image 30b formed by reflection of the light from the light source 110b on the cornea changes according to the distance from the light sources 110a and 110b to the driver's face.

[0053] Therefore, by correcting the threshold value compared with the distance or area based on the distance from the light sources 110a and 110b to the driver's face, the accuracy in identifying the corneal reflection image based on the distance or area is further improved. For example, the threshold correction unit 152h corrects the first threshold value described above to a larger value as the distance from the light sources 110a and 110b to the driver's face is longer. Also, the threshold correction unit 152h corrects the third threshold value described above to a larger value as the distance from the light sources 110a and 110b to the driver's face is longer. The threshold correction unit 152h corrects these threshold values based on a map or table or the like that prescribes in advance the relationship between the distance to the driver's face and these threshold values. At this time, the threshold correction unit 152h may estimate the distance from the light sources 110a and 110b to the driver's face based on a map or table or the like that prescribes in advance the relationship between the position of the seat on which the driver is seated and the distance to the driver's face, for example, based on the position of the seat. Also, when the camera 120 is composed of a stereo camera, the threshold correction unit 152h may calculate the distance from the driver's face from the parallax of the left and right images. The threshold correction unit 152h may calculate in advance the distance from the light sources 110a and 110b to the driver's face, for example, at the time when the driver is seated, etc., before the start of the process of identifying the corneal reflection image.

[0054] The gaze detection unit 152i of the processor 152 detects the driver's gaze using the corneal reflection image identified by the corneal reflection image identification unit 152g by using the above-described known method (corneal reflection method).

[0055] As a result of the line-of-sight detection unit 152i detecting the driver's line of sight, for example, if the driver's line of sight has not been facing forward for a predetermined period or more, there is a possibility that the driver is looking away. Therefore, the processor 152 of the control device 150 issues a command to display a warning on the display device 130. As a result, a warning is displayed on the display device 130. Also, the processor 152 of the control device 150 causes an audio warning to be output from the speaker 140. For example, a warning such as "Please look ahead!" is displayed on the display device 130 and also emitted from the speaker 140. This can prompt the driver who may be looking away to direct their line of sight forward.

[0056] Figure 9 is a flowchart showing the processing performed by the processor 152 of the control device 150 for each predetermined control cycle. First, an image generated by the camera 120 photographing the driver is acquired (step S10). Next, the feature extraction unit 152a of the processor 152 determines whether the driver's face is represented in the image (step S12). If the driver's face is represented in the image, the feature extraction unit 152a determines whether the pupils of the driver's eyes are represented in the image (step S14). If the driver's face is not represented in the image in step S12, or if the pupils of the driver's eyes are not represented in the image in step S14, the process returns to step S10.

[0057] If the pupils of the driver's eyes are represented in the image in step S14, it is determined whether to select a combination of candidates for the corneal reflection image (step S16). If a combination of candidates for the corneal reflection image is to be selected, the candidate selection unit 152c selects a combination of two reflection images that are candidates for the corneal reflection image (step S18).

[0058] On the other hand, as described above, when there is clearly no specular reflection image in the image, it is not necessary to perform the process of specifying the corneal reflection image based on the distance between two points of the reflection image, the angle of the line connecting the two points of the reflection image, or the area of the reflection image by selecting a combination of candidates for the corneal reflection image. Therefore, in such a case, it is determined that a combination of candidates for the corneal reflection image is not selected in step S16, and the process proceeds to step S32, where the line-of-sight detection is performed based on a predetermined corneal reflection image among the two corneal reflection images.

[0059] Next, the distance calculation unit 152d calculates the distance between two reflection image candidates, and the corneal reflection image specifying unit 152g determines whether the distance between the candidates is equal to or less than a first threshold (step S20). When the distance between the candidates is equal to or less than the first threshold, the angle calculation unit 152e calculates the angle formed by the vector between the two reflection image candidates and the horizontal axis, and the corneal reflection image specifying unit 152g determines whether the angle formed by the candidate vector and the horizontal axis is equal to or less than a second threshold (step S22).

[0060] When the angle formed by the candidate vector and the horizontal axis is equal to or less than the second threshold, the area calculation unit 152f calculates the areas of the two reflection image candidates, and the corneal reflection image specifying unit 152g determines whether the areas of the two reflection image candidates are equal to or less than a third threshold (step S24). When the areas of the two reflection image candidates are equal to or less than the third threshold, it is determined whether all combinations of candidates have been selected (step S26). When all combinations of candidates have been selected, the corneal reflection image specifying unit 152g specifies a combination of corneal reflection images from among the combinations of candidates for the corneal reflection image (step S28). Specifically, the corneal reflection image specifying unit 152g specifies two reflection images whose inter-candidate distance is equal to or less than the first threshold, the angle formed by the candidate vector and the horizontal axis is equal to or less than the second threshold, and the area is equal to or less than the third threshold as the combination of corneal reflection images.

[0061] Next, the corneal reflection image specifying unit 152g specifies one corneal reflection image included in the combination of corneal reflection images specified in step S26 as the corneal reflection image to be used for line-of-sight detection (step S30).

[0062] For example, in the example of FIG. 8, when the corneal reflection images 30a and 30b are selected as candidates, if the distance between the corneal reflection image 30a and the corneal reflection image 30b is L4, the area of the corneal reflection image 30b is S1, the area of the corneal reflection image 30a is S2, the distance threshold is L, the angle threshold is θ, and the area threshold is S, then since the conditions of L4≦L, θ1≦θ, and S1≦S and S2≦S are satisfied, the corneal reflection images 30a and 30b are specified as a combination of true corneal reflection images, and furthermore, either the corneal reflection image 30a or the corneal reflection image 30b is specified as the corneal reflection image to be used for line-of-sight detection. On the other hand, when the corneal reflection image 30a and the glasses reflection image 44 are selected as candidates, since at least one of the distance, angle, and area does not satisfy the threshold, the corneal reflection image 30a and the glasses reflection image 44 are not specified as a combination of corneal reflection images.

[0063] If any of the conditions in steps S20, S22, S24, and S26 is not satisfied, the process returns to step S18, and the candidate selection unit 152c selects a combination of two reflection images that are candidates for a new corneal reflection image, and the processes after step S20 are performed again.

[0064] When the corneal reflection image to be used for line-of-sight detection is specified in step S30, the line-of-sight detection unit 152i performs line-of-sight detection based on the specified corneal reflection image (step S32). After step S32, the processing in this control cycle ends.

[0065] Note that in the process of FIG. 9, the determination processes are performed in the order of determination based on distance (step S20), determination based on angle (step S22), and determination based on area (step S24), but the order of determination is not limited to this, and the determination may be performed in an order different from that of FIG. 9. Also, these determination processes may be performed simultaneously (in parallel).

[0066] As described above, according to the present embodiment, by projecting light from the light projecting unit 110 having two light sources separated from each other toward the driver's eyes, two corneal reflection images appear for one eye. Then, by excluding the spectacle reflection image based on the distance between the two points of the reflection image, the angle of the line connecting the two points of the reflection image, and the area of the reflection image, it is possible to suppress the spectacle reflection image from being erroneously detected as the corneal reflection image. Thereby, even in the case of a spectacle wearer, more accurate gaze detection becomes possible.

[0067] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the claims.

Explanation of Reference Numerals

[0068] 10 Eye 12 Contour 20 Pupil 30a, 30b Corneal reflection image 40, 42, 44 Spectacle reflection image 50 Image 100 Occupant support system 110 Light projecting unit 110a, 110b Light source 120 Camera 130 Display device 140 Speaker 150 Control device 152 Processor 152a Feature extraction unit 152b Reflection image extraction unit 152c Candidate selection unit 152d Distance calculation unit 152e Angle calculation unit 152f Area calculation unit 152g Corneal reflection image identification unit 152h Threshold correction unit 152i Gaze detection unit 154 Memory 156 Communication interface 160 Camera unit

Claims

1. A light projecting unit that projects light toward an eye from two light sources spaced apart from each other; An imaging unit that images the illuminated eye; A distance calculation unit that calculates the distance between two of a plurality of reflected images obtained by reflection of the light projected from the light source in the vicinity of the eye, for each combination of two of the reflected images; A corneal reflection image specifying unit that specifies a corneal reflection image used for gaze detection, excluding a reflection image reflected by an object covering the eye from the light projected from the light source, based on a predetermined characteristic value; Comprising: The predetermined characteristic value includes the distance calculated by the distance calculation unit, a corneal reflection image specifying device.

2. The corneal reflection image specifying unit excludes combinations of two reflected images in which the distance exceeds a first threshold value, and specifies, as the corneal reflection image used for gaze detection, one of the reflected images included in combinations of two reflected images in which the distance is equal to or less than the first threshold value. The corneal reflection image specifying device according to claim 1.

3. Comprising an angle calculation unit that calculates, for each combination of two reflected images, an angle between a straight line connecting the two reflected images and a straight line connecting the two light sources when viewed from the imaging direction by the imaging unit; The predetermined characteristic value includes the angle calculated by the angle calculation unit; The corneal reflection image specifying unit excludes combinations of two reflected images in which the angle exceeds a second threshold value, and specifies, as the corneal reflection image used for gaze detection, one of the reflected images included in combinations of two reflected images in which the angle is equal to or less than the second threshold value. The corneal reflection image specifying device according to claim 2.

4. Comprising an area calculation unit that calculates the area of the reflected image for each combination of two reflected images; The predetermined characteristic value includes the area calculated by the area calculation unit; The corneal reflection image specifying unit excludes combinations of two reflected images in which the area exceeds a third threshold value, and specifies, as the corneal reflection image used for gaze detection, one of the reflected images included in combinations of two reflected images in which the area is equal to or less than the third threshold value. The corneal reflection image specifying device according to claim 2 or 3.

5. The corneal reflection image specifying device according to claim 2, comprising a threshold correction unit that corrects the first threshold value to a larger value as the distance from the light source to the face increases.

6. The corneal reflection image specifying device according to claim 4, comprising a threshold correction unit that corrects the third threshold value to a larger value as the distance from the light source to the face increases.

7. An eye-gaze detection device comprising an eye-gaze detection unit that performs eye-gaze detection by the corneal reflection method based on the corneal reflection image specified by the corneal reflection image specifying device according to any one of claims 1, 2, 3, and 5.

Images