Gaze detection device, gaze detection method, and gaze detection program

Abstract

To provide a visual line detection device, a visual line detection method, and a visual line detection program capable of suppressing deterioration in detection accuracy.SOLUTION: A visual line detection device includes: a display unit for displaying an image; a plurality of light sources for emitting detection light and radiating it onto at least one eyeball of a subject; an imaging unit for capturing an image of the eyeball irradiated with the detection light; a position detection unit for detecting the position of a pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of a corneal reflection center indicating the center of the corneal reflection from the captured image; a fixation point detection unit for calculating a position of a fixation point of the subject on the basis of the position of the pupil center and the position of the corneal reflection center; and a light source control unit for switching the light sources for emitting the detection light of the plurality of light sources on the basis of whether a shape of a reflection image of the detection light in the eyeball image is included in a reference shape.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a gaze detection device, a gaze detection method, and a gaze detection program. [Background technology]

[0002] A gaze detection device is known that emits detection light from a light source and irradiates it onto the eyeball of a subject, acquires an image of the eyeball irradiated with the detection light, calculates the pupil center and the corneal curvature center based on the image of the pupil and the reflected image of the detection light in the acquired image, and detects the vector from the corneal curvature center to the pupil center as the gaze direction of the subject (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4649319 Summary of the Invention [Problem to be solved by the invention]

[0004] In the gaze detection device described above, the reflected image of the detection light irradiated onto the subject's eyeball may be present on the boundary between the cornea and the sclera or on the sclera, which may result in a decrease in gaze detection accuracy because the cornea and the sclera have different radii of curvature.

[0005] The present invention has been made in view of the above, and has an object to provide a gaze detection device, a gaze detection method, and a gaze detection program that are capable of suppressing a decrease in detection accuracy. [Means for solving the problem]

[0006] The gaze detection device of the present invention comprises a display unit that displays an image, a plurality of light sources that emit detection light and irradiate it onto at least one eyeball of a subject, an imaging unit that captures an image of the eyeball irradiated with the detection light, a position detection unit that detects from the captured image the position of the pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, a gaze point detection unit that calculates the position of the gaze point of the subject based on the position of the pupil center and the position of the corneal reflex center, and a light source control unit that switches the light source that emits the detection light from among the plurality of light sources based on whether the shape of the reflected image of the detection light in the image of the eyeball is included in a reference shape.

[0007] The gaze detection method of the present invention includes displaying an image on a display unit, emitting detection light from a plurality of light sources and irradiating it onto at least one eyeball of a subject, capturing an image of the eyeball irradiated with the detection light, detecting from the captured image the position of the pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, calculating the position of the subject's point of gaze based on the position of the pupil center and the position of the corneal curvature center, and switching the light source that emits the detection light from among the plurality of light sources based on whether the shape of the reflected image of the detection light in the image of the eyeball is included in a reference shape.

[0008] The gaze detection program of the present invention causes a computer to execute the following processes: displaying an image on a display unit; emitting detection light from multiple light sources and irradiating it onto at least one eyeball of a subject; capturing an image of the eyeball irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's gaze point based on the position of the pupil center and the position of the corneal curvature center; and switching the light source that emits the detection light from among the multiple light sources based on whether the shape of the reflected image of the detection light in the image of the eyeball is included in a reference shape. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a gaze detection device, a gaze detection method, and a gaze detection program that are capable of suppressing a decrease in detection accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a perspective view schematically illustrating an example of a gaze detection device according to this embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of the gaze detection device according to this embodiment. [Figure 3] FIG. 3 is a functional block diagram showing an example of the gaze detection device according to the present embodiment. [Figure 4] FIG. 4 is a diagram showing an example in which the eyeball is illuminated by the first light source and the second light source. [Figure 5] FIG. 5 is a schematic diagram for explaining the principle of the calibration process according to this embodiment. [Figure 6] FIG. 6 is a schematic diagram for explaining the principle of the gaze detection process according to this embodiment. [Figure 7] FIG. 7 is a diagram showing an example of an eyeball on which a reflected image of detection light is formed. [Figure 8] FIG. 8 is a diagram showing an example of an eyeball on which a reflected image of detection light is formed. [Figure 9] FIG. 9 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. [Figure 10] FIG. 10 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. [Figure 11] FIG. 11 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. [Figure 12] FIG. 12 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. [Figure 13] FIG. 13 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. [Figure 14]FIG. 14 is a flowchart showing an example of the gaze detection process in the gaze detection method according to this embodiment. [Figure 15] FIG. 15 is a diagram showing another example of an eyeball on which a reflected image of detection light is formed. [Figure 16] FIG. 16 is a diagram showing another example of the operation of the lighting device in the gaze detection process. [Figure 17] FIG. 17 is a diagram showing another example of the operation of the lighting device in the gaze detection process. [Figure 18] FIG. 18 is a diagram showing another example of the operation of the lighting device in the gaze detection process. [Figure 19] FIG. 19 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment. [Figure 20] FIG. 20 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of a gaze detection device, a gaze detection method, and a gaze detection program according to the present invention will be described with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical.

[0012] In the following explanation, the positional relationship of each part will be explained using a three-dimensional global coordinate system. The direction parallel to the first axis of a predetermined plane is defined as the X-axis direction, the direction parallel to the second axis of the predetermined plane that is perpendicular to the first axis is defined as the Y-axis direction, and the direction parallel to the third axis that is perpendicular to both the first and second axes is defined as the Z-axis direction. The predetermined plane includes the XY plane.

[0013] (Gaze detection device) 1 is a perspective view schematically illustrating an example of a gaze detection device 100 according to this embodiment. As shown in FIG. 1, the gaze detection device 100 includes a display unit 101, a stereo camera device 102, and an illumination device 103.

[0014] The display unit 101 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). In this embodiment, the display unit 101 displays an image. In this embodiment, the display unit 101 displays, for example, an index for evaluating the visual function of a subject. The display unit 101 is substantially parallel to the XY plane. The X-axis direction is the left-right direction of the display unit 101, the Y-axis direction is the up-down direction of the display unit 101, and the Z-axis direction is the depth direction perpendicular to the display unit 101.

[0015] The stereo camera device 102 has a first camera 102A and a second camera 102B. The stereo camera device 102 is disposed below the display unit 101. The first camera 102A and the second camera 102B are disposed in the X-axis direction. The first camera 102A is disposed in the -X direction from the second camera 102B. The first camera 102A and the second camera 102B each include an infrared camera, and have an optical system that can transmit near-infrared light with a wavelength of, for example, 850 [nm], and an imaging element that can receive the near-infrared light.

[0016] The illumination device (light source) 103 has a first light source (lower first light source) 103A, a second light source (lower second light source) 103B, a third light source (upper light source) 103C, and a fourth light source (upper light source) 103D. The first light source 103A and the second light source 103B are disposed below the display unit 101. The third light source 103C and the fourth light source 103D (upper light source) are disposed above the display unit 101.

[0017] First light source 103A and second light source 103B are arranged in the X-axis direction. First light source 103A is arranged in the -X direction from first camera 102A. Second light source 103B is arranged in the +X direction from second camera 102B. Third light source 103C and fourth light source 103D are arranged in the X-axis direction. Third light source 103C is arranged in the -X direction from first camera 102A. Fourth light source 103D is arranged in the +X direction from second camera 102B.

[0018] First light source 103A, second light source 103B, third light source 103C, and fourth light source 103D each include an LED (light emitting diode) light source and are capable of emitting near-infrared light with a wavelength of 850 [nm], for example. First light source 103A and second light source 103B may be disposed between first camera 102A and second camera 102B. Similarly, third light source 103C and fourth light source 103D may be disposed between first camera 102A and second camera 102B in the X direction. Stereo camera device 102 may be disposed above display unit 101.

[0019] Illumination device 103 emits near-infrared light, which is detection light, to illuminate eyeball 111 of the subject. Stereo camera device 102 captures an image of part of eyeball 111 (hereinafter, this will be collectively referred to as "eyeball") with second camera 102B when detection light emitted by first light source 103A or third light source 103C is irradiated onto eyeball 111, and captures an image of eyeball 111 with first camera 102A when detection light emitted by second light source 103B or fourth light source 103D is irradiated onto eyeball 111.

[0020] A frame synchronization signal is output from at least one of first camera 102A and second camera 102B. First light source 103A, second light source 103B, third light source 103C, and fourth light source 103D emit detection light based on the frame synchronization signal. First camera 102A captures image data of eyeball 111 when detection light emitted from second light source 103B or fourth light source 103D is irradiated onto eyeball 111. Second camera 102B captures image data of eyeball 111 when detection light emitted from first light source 103A or third light source 103C is irradiated onto eyeball 111.

[0021] When the eyeball 111 is irradiated with detection light, part of the detection light is reflected by the pupil 112, and the light from the pupil 112 enters the stereo camera device 102. In addition, when the eyeball 111 is irradiated with detection light, a corneal reflection image 113, which is a virtual image of the cornea, is formed on the eyeball 111, and the light from the corneal reflection image 113 enters the stereo camera device 102.

[0022] By appropriately setting the relative positions of the first camera 102A and the second camera 102B and the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D, the intensity of light incident on the stereo camera device 102 from the pupil 112 is reduced, and the intensity of light incident on the stereo camera device 102 from the corneal reflection image 113 is increased. That is, the image of the pupil 112 captured by the stereo camera device 102 has low brightness, and the image of the corneal reflection image 113 has high brightness. The gaze detection device 100 can detect the position of the pupil 112 and the position of the corneal reflection image 113 based on the brightness of the captured images.

[0023] 2 is a diagram showing an example of the hardware configuration of the gaze detection device 100 according to this embodiment. As shown in Fig. 2, the gaze detection device 100 includes a display unit 101, a stereo camera device 102, an illumination device 103, a computer system (control unit) 20, an input / output interface device 30, a drive circuit 40, an output device 50, and an input device 60.

[0024] The computer system 20, the drive circuit 40, the output device 50, and the input device 60 communicate with each other via an input / output interface device 30. The computer system 20 includes an arithmetic processing unit 20A and a storage device 20B. The arithmetic processing unit 20A includes a microprocessor such as a central processing unit (CPU). The storage device 20B includes memory or storage such as a read only memory (ROM) and a random access memory (RAM). The arithmetic processing unit 20A performs arithmetic processing in accordance with a computer program 20C stored in the storage device 20B.

[0025] The driving circuit 40 generates driving signals and outputs them to the display unit 101, the stereo camera device 102, and the lighting device 103. The driving circuit 40 also supplies image data of the eyeball 111 captured by the stereo camera device 102 to the computer system 20 via the input / output interface device 30.

[0026] The output device 50 includes a display unit such as a flat panel display. The output device 50 may also include a printing device. The input device 60 generates input data when operated. The input device 60 includes a keyboard or a mouse for a computer system. The input device 60 may also include a touch sensor provided on the display screen of the output device 50, which is the display unit.

[0027] In this embodiment, the display unit 101 and the computer system 20 are separate devices. However, the display unit 101 and the computer system 20 may be integrated. For example, if the gaze detection device 100 includes a tablet-type personal computer, the computer system 20, the input / output interface device 30, the drive circuit 40, and the display unit 101 may be mounted on the tablet-type personal computer.

[0028] 3 is a functional block diagram showing an example of the gaze detection device 100 according to this embodiment. As shown in FIG. 3, the input / output interface device 30 has an input / output unit 302. The drive circuit 40 has a display device drive unit 402 that generates a drive signal for driving the display unit 101 and outputs it to the display unit 101, a first camera input / output unit 404A that generates a drive signal for driving the first camera 102A and outputs it to the first camera 102A, a second camera input / output unit 404B that generates a drive signal for driving the second camera 102B and outputs it to the second camera 102B, and a light source drive unit 406 that generates drive signals for driving the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D and outputs them to the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D. Furthermore, first camera input / output unit 404A supplies image data of eyeball 111 photographed by first camera 102A to computer system 20 via input / output unit 302. Second camera input / output unit 404B supplies image data of eyeball 111 photographed by second camera 102B to computer system 20 via input / output unit 302.

[0029] The computer system 20 controls the gaze detection device 100. The computer system 20 includes a display control unit 21, a light source control unit 22, an image data acquisition unit 23, a position detection unit 24, a curvature center calculation unit 25, a gaze point detection unit 26, a memory unit 27, and an input / output control unit 28. The functions of the computer system 20 are realized by an arithmetic processing unit 20A and a memory device 20B.

[0030] The display control unit 21 causes the display unit 101 to display an image to be shown to the subject. The display control unit 21 can display, for example, a target image in the calibration process at a plurality of positions (target positions) on the display unit 101. The display control unit 21 may sequentially switch the target image to be displayed at a plurality of target positions one by one, or may display the target image so that it moves sequentially to a plurality of target positions within the display unit 101. The number of target positions at which the target image is to be displayed can be set, for example, by an operator inputting the number using the input device 60 or the like.

[0031] The light source control unit 22 controls the light source driving unit 406 to control the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D to emit or not emit light. The light source control unit 22 switches between the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D to emit detection light based on the target distance between the pupil center and the corneal reflex center. Examples of the target distance include the vertical distance between the pupil center and the corneal reflex center, the horizontal distance between the pupil center and the corneal reflex center, and the shortest distance between the pupil center and the corneal reflex center. When the target distance between the pupil center and the corneal reflex center is equal to or greater than a predetermined threshold while the detection light is being emitted from the first light source 103A and the second light source 103B, the light source control unit 22 can switch to emitting detection light from the third light source 103C and the fourth light source 103D. In addition, when detection light is emitted from the third light source 103C and the fourth light source 103D, the light source control unit 22 can switch to emitting detection light from the first light source 103A and the second light source 103B when the target distance between the pupil center and the corneal reflex center is equal to or greater than a predetermined threshold.

[0032] Furthermore, the light source control unit 22 switches the light source that emits the detection light from among the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D, based on whether the shape of the reflected image of the detection light in the image of the eyeball 111 is included in the reference shape. In the present embodiment, the reference shape may be a single ellipse. When the light source control unit 22 determines that the shape of the reflected image of the detection light in the image of the eyeball 111 is not included in the reference shape while the detection light is being emitted from the first light source 103A and the second light source 103B, the light source control unit 22 can switch to emitting the detection light from the third light source 103C and the fourth light source 103D. When the light source control unit 22 determines that the shape of the reflected image of the detection light in the image of the eyeball 111 is not included in the reference shape while the detection light is being emitted from the third light source 103C and the fourth light source 103D, the light source control unit 22 can switch to emitting the detection light from the first light source 103A and the second light source 103B.

[0033] The image data acquisition unit 23 acquires image data of the subject's eyeball 111 captured by the stereo camera device 102 including the first camera 102A and the second camera 102B from the stereo camera device 102 via the input / output unit 302.

[0034] The position detection unit 24 detects position data of the pupil center based on the image data of the eyeball 111 acquired by the image data acquisition unit 23. The position detection unit 24 also detects position data of the corneal reflex center based on the image data of the eyeball 111 acquired by the image data acquisition unit 23. The pupil center is the center of the pupil 112. The corneal reflex center is the center of the corneal reflection image 113. The position detection unit 24 detects position data of the pupil center and position data of the corneal reflex center for each of the left and right eyeballs 111 of the subject.

[0035] The curvature center calculation unit 25 calculates position data of the corneal curvature center of the eyeball 111 based on the image data of the eyeball 111 acquired by the image data acquisition unit 23 .

[0036] The gaze point detection unit 26 detects position data of the gaze point of the subject based on image data of the eyeballs 111 acquired by the image data acquisition unit 23. In this embodiment, the gaze point position data refers to position data of the intersection between the gaze vector of the subject defined in a three-dimensional global coordinate system and the display unit 101. The gaze point detection unit 26 detects the gaze vectors of each of the subject's left and right eyeballs 111 based on position data of the pupil center and position data of the corneal curvature center acquired from the image data of the eyeballs 111. After the gaze vectors are detected, the gaze point detection unit 26 detects position data of the gaze points indicating the intersection between the gaze vector and the display unit 101.

[0037] The storage unit 27 stores various data and programs related to the above-mentioned gaze detection. For example, the storage unit 27 can store data on images to be displayed on the display unit 101 for each color and brightness of the background image. The storage unit 27 also stores position data of the gaze point calculated in each calibration process.

[0038] The memory unit 27 also stores a gaze detection program that causes the computer to execute the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature; and switching the light source that emits detection light from among the multiple light sources based on the target distance between the pupil center and the corneal reflex center.

[0039] The memory unit 27 also stores a gaze detection program that causes the computer to execute the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's point of gaze based on the position of the pupil center and the position of the center of corneal curvature; and switching the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflex image 113 of the detection light in the image of the eyeball 111 is included in a reference shape.

[0040] The input / output control unit 28 outputs data to at least one of the display unit 101 and the output device 50 .

[0041] Next, an overview of the processing of the center of curvature calculation unit 25 according to this embodiment will be described. In this embodiment, a case will be described in which the eyeball 111 is illuminated by the first light source 103A and the second light source 103B, and the eyeball 111 is photographed by two cameras, the first camera 102A and the second camera 102B. Note that the number of light sources and cameras is not limited to two, and a similar explanation can be given for a case in which there is one light source and one camera. Below, the principle of the gaze detection method according to this embodiment will be described.

[0042] Fig. 4 is a diagram showing an example in which eyeball 111 is illuminated by first light source 103A and second light source 103B. As shown in Fig. 4, in this embodiment, first camera 102A and second light source 103B, and second camera 102B and first light source 103A are disposed at positions symmetrical to each other with respect to a line passing through the midpoint between first camera 102A and second camera 102B. It can be considered that a virtual light source (reference position of the light source) 103V exists at the midpoint between first camera 102A and second camera 102B.

[0043] Corneal reflection center 121 indicates the corneal reflection center in the image of eyeball 111 photographed by second camera 102B. Corneal reflection center 122 indicates the corneal reflection center in the image of eyeball 111 photographed by first camera 102A. Corneal reflection center 124 indicates the corneal reflection center corresponding to virtual light source 103V.

[0044] The position data of the corneal reflection center 124 is calculated based on the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 photographed by the stereo camera device 102. The stereo camera device 102 detects the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 in a three-dimensional local coordinate system defined for the stereo camera device 102. Camera calibration of the stereo camera device 102 is performed in advance using a stereo calibration method, and transformation parameters for transforming the three-dimensional local coordinate system of the stereo camera device 102 into a three-dimensional global coordinate system are calculated. The transformation parameters are stored in the storage unit 27. The center of curvature calculation unit 25 converts the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 photographed by the stereo camera device 102 into position data in the three-dimensional global coordinate system using the transformation parameters. The curvature center calculation unit 25 calculates the position data of the corneal reflection center 124 in the three-dimensional global coordinate system based on the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 defined in the three-dimensional global coordinate system.

[0045] Corneal center of curvature 110 exists on a straight line 123 connecting virtual light source 103V and corneal reflection center 124. Center of curvature calculation unit 25 calculates the position on straight line 123 at a predetermined distance from corneal reflection center 124 as the position of corneal center of curvature 110. Corneal radius of curvature 109 is used as the predetermined value. Corneal radius of curvature 109 is the distance between the corneal surface and corneal center of curvature 110. The value of corneal radius of curvature 109 can be a predetermined value selected from, for example, common corneal radius of curvature values.

[0046] When the third light source 103C and the fourth light source 103D are used, the third light source 103C and the fourth light source 103D are arranged above the display unit 101 so that the X coordinates of the third light source 103C and the fourth light source 103D are the same as those of the first light source 103A and the second light source 103B, and the corneal curvature center 110 is calculated in the same manner as above by assuming that a virtual light source 103V2 (not shown) exists at an intermediate position between the third light source 103C and the fourth light source 103D.

[0047] [Gaze detection method] In the gaze detection method according to this embodiment, a calibration process is performed, followed by a gaze point detection process. First, the principle of the calibration process will be described. FIG. 5 is a schematic diagram for explaining the principle of the calibration process according to this embodiment. In the calibration process, a target position 130 is set to be gazed at by the subject. The target position 130 is defined in a three-dimensional global coordinate system. The display control unit 21 displays a target image at the set target position 130.

[0048] First light source 103A and second light source 103B illuminate eyeball 111. First camera 102A and second camera 102B photograph eyeball 111. For example, when detection light is emitted from first light source 103A, second camera 102B photographs eyeball 111. When detection light is emitted from second light source 103B, first camera 102A photographs eyeball 111. When third light source 103C and fourth light source 103D are used instead of first light source 103A and second light source 103B, detection light is emitted from third light source 103C and fourth light source 103D, and eyeball 111 is photographed at different times by first camera 102A and second camera 102B, for example. The position detection unit 24 detects position data of the pupil center 112C and the corneal reflex center 113C based on the image data of the eyeball 111 acquired by the image data acquisition unit 23. The position detection unit 24 converts each of the detected position data into a global coordinate system.

[0049] When detection light is emitted from first light source 103A and second light source 103B, curvature center calculation unit 25 calculates position data of corneal center of curvature 110 based on position data of virtual light source 103V, position data of target position 130, position data of pupil center 112C, and position data of corneal reflection center 113C. Specifically, curvature center calculation unit 25 obtains first straight line 141 connecting virtual light source 103V and corneal reflection center 113C. When detection light is emitted from third light source 103C and fourth light source 103D, curvature center calculation unit 25 calculates position data of corneal center of curvature 110 based on position data of virtual light source 103V2, position data of target position 130, position data of pupil center 112C, and position data of corneal reflection center 113C.

[0050] The curvature center calculation unit 25 also calculates a second straight line 142 connecting the target position 130 and the pupil center 112C. The curvature center calculation unit 25 calculates the intersection of the first straight line 141 and the second straight line 142 as position data of the corneal curvature center 110. The curvature center calculation unit 25 then calculates a distance 127 between the corneal curvature center 110 and the pupil center 112C and stores the distance 127 in the storage unit 27 as calibration data. In this embodiment, the curvature center calculation unit 25 calculates the distance (Ra) between the corneal curvature center 110 and the pupil center 112C when the first light source 103A and the second light source 103B emit detection light, and the distance (Rb) between the corneal curvature center 110 and the pupil center 112C when the third light source 103C and the fourth light source 103D emit detection light.

[0051] Next, the principle of the gaze detection process will be described. Fig. 6 is a schematic diagram for explaining the principle of the gaze detection process according to this embodiment. In the gaze detection process, as in the calibration process, the eyeball 111 is illuminated using the first light source 103A and the second light source 103B, or the third light source 103C and the fourth light source 103D. The first camera 102A and the second camera 102B capture images of the eyeball 111. The position detection unit 24 detects position data of the pupil center 112C and the corneal reflex center 113C based on image data of the eyeball 111 acquired by the image data acquisition unit 23.

[0052] When detection light is emitted using first light source 103A and second light source 103B, curvature center calculation unit 25 calculates position data of corneal center of curvature 110 based on position data of virtual light source 103V, position data of pupil center 112C, position data of corneal reflection center 113C, and distance 127 between corneal center of curvature 110 and pupil center 112C calculated in the calibration process. Specifically, curvature center calculation unit 25 obtains straight line 173 connecting virtual light source 103V and corneal reflection center 113C. Similarly, when detection light is irradiated using the third light source 103C and the fourth light source 103D, the curvature center calculation unit 25 calculates the position data of the corneal curvature center 110 based on the position data corresponding to the virtual light source 103V2, the position data of the pupil center 112C, the position data of the corneal reflection center 113C, and the distance 127 between the corneal curvature center 110 and the pupil center 112C calculated in the calibration process.

[0053] Furthermore, the curvature center calculation unit 25 obtains, as position data of the corneal curvature center 110, a position that is a distance equivalent to the distance 127 away from the pupil center 112C toward the inside of the eyeball 111. The gaze point detection unit 26 obtains a straight line 178 that connects the pupil center 112C and the corneal curvature center 110, and calculates position data of the intersection 166 of the straight line 178 and the display unit 101 as position data of the gaze point.

[0054] 7 is a diagram showing an example of an eyeball on which a reflection image of the detection light is formed. As shown in the upper part of FIG. 7, when the subject looks at a position not far from virtual light source 103V, for example, and corneal reflection image 113 of the detection light irradiated onto subject's eyeball 111 is present on cornea 111a, the shape of corneal reflection image 113 becomes, for example, a single ellipse with a small oblateness, and therefore position data of the corneal reflection center can be detected with high accuracy.

[0055] As shown in the middle and bottom panels of Figure 7, when the subject looks at a position far away from virtual light source 103V, corneal reflection image 113 of detection light irradiated onto subject's eyeball 111 may be present at the boundary between cornea 111a and sclera 111b. In this case, due to differences in the radius of curvature and reflectance, the shape of corneal reflection image 113 of detection light may be distorted, for example, as shown in the middle panel of Figure 7, becoming an ellipse or oval with a large flattening ratio, or multiple corneal reflection images 113 may be formed, as shown in the bottom panel of Figure 7. This may reduce the accuracy of detecting position data of the corneal reflection center, thereby reducing the accuracy of detecting the gaze.

[0056] In contrast, in this embodiment, as an example, based on the target distance between the pupil center and the corneal reflection center of the subject, control is performed to switch the light source that emits the detection light so that the reflected image of the detection light is formed at a position that does not extend beyond the cornea 111a of the subject. The position at which the reflected image of the detection light is formed is determined by, for example, the position of the subject's eyeball and the positional relationship between the camera and the light source.

[0057] FIG. 8 is a diagram illustrating an example of an eyeball on which a reflected image of the detection light is formed. As illustrated in FIG. 8, the light source control unit 22 calculates the target distance between the pupil center 112C and the corneal reflex center 113C, which is calculated based on image data of the eyeball 111. The light source control unit 22 can calculate the target distance, for example, in the vertical direction (Y direction) and the horizontal direction (X direction) between the pupil center 112C and the corneal reflex center 113C. The light source control unit 22 may also calculate the shortest distance between the pupil center 112C and the corneal reflex center 113C as the target distance. Hereinafter, the vertical distance between the pupil center 112C and the corneal reflex center 113C will be referred to as the target distance DY, and the horizontal distance will be referred to as the target distance DX. In the example illustrated in FIG. 8, the light source control unit 22 calculates the target distance DY and the target distance DX. The light source control unit 22 can calculate, as the object distance DY, the absolute value |Y2-Y1| of the difference between the Y coordinate (Y2) of the pupil center 112C and the Y coordinate (Y1) of the corneal reflection center 113C. The light source control unit 22 can also calculate, as the object distance DX, the absolute value |X2-X1| of the difference between the X coordinate (X2) of the pupil center 112C and the X coordinate (X1) of the corneal reflection center 113C.

[0058] In this embodiment, when the subject moves the eyeball 111, the positions of the pupil center 112C, the cornea 111a, and the sclera 111b change. As a result, the relative positional relationship between the corneal reflection image 113 and the pupil center 112C, the cornea 111a, and the sclera 111b changes, and the object distances DY and DX change. For example, as shown in FIG. 8, when the corneal reflection image 113 is located to the lower right of the pupil center 112C, if the subject moves the eyeball 111 opposite the corneal reflection image 113 (upper left: dashed dotted line in the figure), the relative positional relationship between the corneal reflection image 113 and the pupil center 112C, the cornea 111a, and the sclera 111b changes, and the object distances DY and DX increase.

[0059] In this embodiment, thresholds for the object distance DY and the object distance DX are set in advance so that the corneal reflection image 113 fits within a position on the cornea 111a. The light source control unit 22 may set the thresholds for the object distance DY and the object distance DX in the calibration process. In this case, for example, values of the object distance DY and the object distance DX when the subject gazes at the center position of the display screen 101S of the display unit 101 can be used as the thresholds. The light source control unit 22 sets thresholds for when the detection light is emitted from the first light source 103A and the second light source 103B, and thresholds for when the detection light is emitted from the third light source 103C and the fourth light source 103D, respectively. Therefore, if the object distance DY exceeds the threshold when the detection light is emitted from the first light source 103A and the second light source 103B, it can be determined that the subject is gazing above the center position of the display screen 101S. Furthermore, when the object distance DY exceeds a threshold value when the detection light is emitted from the third light source 103C and the fourth light source 103D, it can be determined that the subject is gazing downward from the center position of the display screen 101S. The light source control unit 22 switches the illumination device 103 that emits the detection light so that the calculated object distance DY and object distance DX are less than a preset threshold value. The light source control unit 22 may control the switching of the illumination device 103 using only one of the object distance DY and the object distance DX. Below, an example will be described in which a threshold value is set for the object distance DY and control is performed so that the object distance DY is less than the threshold value. The threshold value of the object distance DY when the detection light is emitted from the first light source 103A and the second light source 103B is set to α, and the threshold value of the object distance DY when the detection light is emitted from the third light source 103C and the fourth light source 103D is set to β. When switching control of the lighting device 103 is performed using both the object distance DY and the object distance DX, control can be performed so that the value of at least one of the object distance DY and the object distance DX is less than a threshold value.

[0060] 9 to 13 are diagrams showing an example of the operation of the illumination device 103 in the gaze detection process. When detecting the first gaze point in the gaze detection process, the light source control unit 22 controls, for example, the first light source 103A and the second light source 103B to emit detection light. As shown in Fig. 9, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is less than the threshold value α, the light source control unit 22 controls the first light source 103A and the second light source 103B, which are bottom light sources, to emit detection light.

[0061] On the other hand, as shown in FIG. 10, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is equal to or greater than the threshold value α, the light source control unit 22 switches to emit detection light from the third light source 103C and the fourth light source 103D, which are upper light sources, as shown in FIG. 11.

[0062] The light source control unit 22 can perform similar control even when detection light is emitted from the third light source 103C and the fourth light source 103D, which are upper light sources. As shown in Fig. 11, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is less than the threshold value β, the light source control unit 22 controls so that detection light is emitted from the third light source 103C and the fourth light source 103D, which are upper light sources.

[0063] On the other hand, as shown in FIG. 12, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is equal to or greater than the threshold value β, the light source control unit 22 switches to emit detection light from the first light source 103A and the second light source 103B, which are lower light sources, as shown in FIG. 13.

[0064] Light source control unit 22 causes one of first light source 103A and second light source 103B, which are lower light sources, to emit light to irradiate eyeball 111 with detection light, and photographs eyeball 111 of the subject using either first camera 102A or second camera 102B, whichever is farther from the emitting light source. Thereafter, light source control unit 22 causes the other of first light source 103A and second light source 103B to emit light to irradiate eyeball 111 with detection light, and photographs eyeball 111 of the subject using either first camera 102A or second camera 102B, whichever is farther from the emitting light source. For example, when detection light is emitted from first light source 103A, eyeball 111 is photographed by second camera 102B. When detection light is emitted from second light source 103B, eyeball 111 is photographed by first camera 102A.

[0065] Similarly, light source control unit 22 causes one of third light source 103C and fourth light source 103D, which are upper light sources, to emit light to irradiate eyeball 111 with detection light, and photographs eyeball 111 of the subject using either first camera 102A or second camera 102B, whichever is farther from the emitting light source. Thereafter, light source control unit 22 causes the other of third light source 103C and fourth light source 103D to emit light to irradiate eyeball 111 with detection light, and photographs eyeball 111 of the subject using either first camera 102A or second camera 102B, whichever is farther from the emitting light source. For example, when detection light is emitted from third light source 103C, eyeball 111 is photographed by second camera 102B. When detection light is emitted from fourth light source 103D, eyeball 111 is photographed by first camera 102A.

[0066] The image data acquisition unit 23 acquires image data. The position detection unit 24 detects position data of the pupil center and the corneal reflex center based on the acquired image data. The position detection unit 24 determines whether the position data of the corneal reflex center has been detected correctly. If the reflected image of the detection light is located on the subject's cornea, the likelihood of detecting a correct value is high. On the other hand, if the reflected image of the detection light is not located on the subject's cornea but is distorted, for example, by extending onto the sclera, the likelihood of detecting a correct value is low. If a correct value is detected, the curvature center calculation unit 25 and the gaze point detection unit 26 perform their respective processes to acquire position data of the gaze point. Note that if a correct value is not detected, the gaze point detection unit 26 may, for example, determine an error in the gaze detection process.

[0067] When a normal value of the corneal reflex center is detected, the curvature center calculation unit 25 calculates the corneal center of curvature based on the detected value. At this time, when the light source emitting the detection light in the gaze detection process is a lower light source (first light source 103A and second light source 103B), the curvature center calculation unit 25 calculates the corneal center of curvature using the value of the distance Ra calculated when the light source emitting the detection light is the lower light source, out of the two types of distances Ra and Rb calculated in the calibration process. Furthermore, when the light source emitting the detection light in the gaze detection process is an upper light source (third light source 103C and fourth light source 103D), the curvature center calculation unit 25 calculates the corneal center of curvature using the value of the distance Rb calculated when the light source emitting the detection light is the upper light source in the calibration process.

[0068] Next, an example of the gaze detection method according to this embodiment will be described with reference to FIG. 14. FIG. 14 is a flowchart showing an example of the gaze detection process in the gaze detection method according to this embodiment. As shown in FIG. 14, in the gaze detection process, the light source control unit 22 causes one of the lower light sources (first light source 103A and second light source 103B) and the upper light source (third light source 103C and fourth light source 103D) to emit detection light (step S101). In this case, the light source control unit 22 controls one of the first light source 103A and the second light source 103B to emit light to irradiate the eyeball 111 with the detection light, and then the eyeball 111 of the subject is photographed by the camera farthest from the emitting light source, either the first camera 102A or the second camera 102B (step S102). Further, under the control of light source control unit 22, the other of first light source 103A and second light source 103B is caused to emit light to irradiate eyeball 111 with detection light, and eyeball 111 of the subject is photographed by either first camera 102A or second camera 102B, whichever is farther from the emitting light source (step S103). For example, when detection light is emitted from first light source 103A, eyeball 111 is photographed by second camera 102B. When detection light is emitted from second light source 103B, eyeball 111 is photographed by first camera 102A.

[0069] The image data acquisition unit 23 acquires image data. The position detection unit 24 detects position data of the pupil center and the corneal reflex center based on the acquired image data. Thereafter, the position data of the gaze point is acquired by the respective processes in the curvature center calculation unit 25 and the gaze point detection unit 26 (step S104). In step S104, when the light source that emits the detection light in the gaze detection process is the lower light source (first light source 103A and second light source 103B), the curvature center calculation unit 25 calculates the corneal center of curvature using the value of the distance Ra calculated when the light source that emits the detection light in the calibration process is the lower light source. In addition, when the light source that emits the detection light in the gaze detection process is the upper light source (third light source 103C), the curvature center calculation unit 25 calculates the corneal center of curvature using the value of the distance Rb calculated when the light source that emits the detection light in the calibration process is the upper light source. Note that if a normal value is not detected in step S104, an error may be determined and the process may be terminated.

[0070] After step S104, the gaze point detection unit 26 determines whether or not to end the gaze point detection (step S105). If the result of the determination in step S105 is that the gaze point detection is to be ended (Yes in step S105), the process ends. On the other hand, if the gaze point detection is not to be ended (No in step S105), the light source control unit 22 determines whether or not the object distance between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is less than a threshold (step S106). If it is determined that the object distance between the pupil center 112C and the corneal reflex center 113C is less than the threshold (Yes in step S106), the light source control unit 22 does not switch the light source that emits the detection light between the lower light source and the upper light source, and causes the processes from step S102 onwards to be repeated (step S107). On the other hand, if it is determined that the target distance between the pupil center 112C and the corneal reflex center 113C is not less than the threshold, the light source control unit 22 switches the light source that emits the detection light between the lower light source and the upper light source, and repeats the processing from step S102 onwards (step S108).

[0071] Next, another example of control for switching the light source that emits the detection light in this embodiment will be described. Hereinafter, the case where control for switching the light source that emits the detection light is performed based on whether or not the shape of the corneal reflection image 113 of the detection light in the image data of the eyeball 111 is included in the reference shape will be described as an example.

[0072] FIG. 15 is a diagram showing another example of an eyeball on which a reflection image of the detection light is formed. As shown in the upper part of FIG. 15, when a corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 is present on the cornea 111a, the shape of the corneal reflection image 113 is, for example, an ellipse with a small oblateness. Therefore, in this embodiment, a single ellipse with an oblateness less than a threshold value can be used as the reference shape. For example, the predetermined value can be calculated in advance when the corneal reflection image 113 is present on the cornea 111a, and the average or minimum value of the calculation results can be used, but this is not limiting. When a corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 is present on the cornea 111a, the shape of the corneal reflection image 113 is included in the reference shape. The reference shape can be stored, for example, in the memory unit 27.

[0073] On the other hand, when the corneal reflection image 113 of the detection light irradiated onto the eyeball 111 of the subject is located at the boundary between the cornea 111a and the sclera 111b, the shape of the corneal reflection image 113 may be, for example, an ellipse or an oval with a large flattening ratio, as shown in the middle of Fig. 15, or multiple corneal reflection images 113 may be formed, as shown in the bottom of Fig. 15. In other words, when the corneal reflection image 113 of the detection light irradiated onto the eyeball 111 of the subject is located at the boundary between the cornea 111a and the sclera 111b, the shape of the corneal reflection image 113 is not included in the reference shape.

[0074] Based on this, the light source control unit 22 performs image processing on the image data of the eyeball 111 to determine the shape of the corneal reflection image 113 of the detection light. For example, the light source control unit 22 calculates an area in the image data where the brightness exceeds a predetermined value as the area of the corneal reflection image 113. The light source control unit 22 calculates the number of calculated areas of the corneal reflection image 113. If the calculation result is 2 or more, the light source control unit 22 can determine that the shape of the corneal reflection image 113 is not included in the reference shape.

[0075] Furthermore, if the calculation result is 1, the light source control unit 22 determines the shape of the calculated corneal reflection image 113. For example, the light source control unit 22 compares the calculated corneal reflection image 113 with the reference shape Q, and if the rate of match is a predetermined value or more, the light source control unit 22 can determine that the shape of the corneal reflection image 113 is included in the reference shape. On the other hand, if the light source control unit 22 compares the calculated corneal reflection image 113 with the reference shape Q and the rate of match is less than a predetermined value, the light source control unit 22 can determine that the shape of the corneal reflection image 113 is not included in the reference shape.

[0076] When the light source control unit 22 determines that the shape of the corneal reflection image 113 is not included in the reference shape, it switches the light source that emits the detection light between the lower light source (first light source 103A, second light source 103B) and the upper light source (third light source 103C, fourth light source 103D). On the other hand, when the light source control unit 22 determines that the shape of the corneal reflection image 113 is included in the reference shape, it does not switch the light source that emits the detection light.

[0077] 16 to 18 are diagrams showing other examples of the operation of the illumination device 103 in the gaze detection process. The light source control unit 22 controls, for example, the first light source 103A and the second light source 103B to emit detection light. For example, it determines whether the shape of the corneal reflection image 113 of the detection light in the image data of the eyeball 111 is included in the reference shape.

[0078] When the light source control unit 22 determines that the shape of the corneal reflection image 113 is included in the reference shape, the light source control unit 22 does not switch the light source that emits the detection light, as shown in Fig. 16. On the other hand, when the light source control unit 22 determines that the shape of the corneal reflection image 113 is not included in the reference shape, as shown in Fig. 17, for example, the light source control unit 22 switches so that the detection light is emitted from the third light source 103C and the fourth light source 103D, which are upper light sources, as shown in Fig. 18.

[0079] Fig. 19 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment. As shown in Fig. 19, steps S201 to S205 are the same as steps S101 to S105 (see Fig. 14) in the case of controlling switching of the light source that emits detection light based on the target distance between the pupil center and the corneal reflex center of the subject.

[0080] As a result of the determination in step S205, if the detection of the gaze point is to be ended (Yes in step S205), the processing is ended. On the other hand, if the detection of the gaze point is not to be ended (No in step S205), the light source control unit 22 determines whether the shape of the corneal reflection image 113 in the image data of the eyeball 111 is included in the reference shape (step S206). If it is determined that the shape of the corneal reflection image 113 is included in the reference shape (Yes in step S206), the light source control unit 22 does not switch the light source that emits the detection light between the lower light source and the upper light source, and causes the processing from step S202 onwards to be repeated (step S207). On the other hand, if it is determined that the shape of the corneal reflection image 113 is not included in the reference shape, the light source control unit 22 switches the light source that emits the detection light between the lower light source and the upper light source, and causes the processing from step S102 onwards to be repeated (step S208).

[0081] Fig. 20 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment. The example shown in Fig. 20 shows a case where control for switching the light source that emits the detection light based on the target distance between the pupil center and the corneal reflex center of the subject is combined with control for switching the light source that emits the detection light based on whether the shape of the corneal reflex image 113 is included in the reference shape.

[0082] As shown in FIG. 20, steps S301 to S304 are similar to steps S101 (S201) to S104 (S204) in the angle control described above (see FIGS. 14 and 19).

[0083] After step S304, the gaze point detection unit 26 determines whether the gaze point has been detected normally (step S305). If it is determined in step S305 that the gaze point has been detected normally (Yes in step S305), the gaze point detection unit 26 and the light source control unit 22 perform the processes of steps S306 to S309. Steps S306 to S309 are similar to the processes of steps S105 to S108 in the control of switching the light source that emits detection light based on the target distance between the pupil center and the corneal reflex center of the subject (see FIG. 14).

[0084] On the other hand, if it is determined in step S305 that the gaze point is not detected normally (No in step S305), the light source control unit 22 performs the processes of steps S310 to S312. Steps S310 to S312 are similar to the processes of steps S206 to S208 in the control of switching the light source that emits the detection light based on whether the shape of the corneal reflection image 113 is included in the reference shape (see FIG. 19).

[0085] Control for switching the light source that emits detection light based on the target distance between the subject's pupil center and corneal reflex center can be performed based on the normal values of the pupil center position and the corneal reflex center position when normal values are calculated for the pupil center position and the corneal reflex center position, that is, when the gaze point position is detected normally. On the other hand, control for switching the light source that emits detection light based on whether the shape of the corneal reflex image 113 is included in the reference shape can be performed based on the shape of the corneal reflex image 113 without calculating the pupil center position and the corneal reflex center position. Therefore, even when normal values cannot be obtained for the pupil center position and the corneal reflex center position, control for switching the light source that emits detection light can be performed appropriately.

[0086] As described above, the gaze detection device 100 of this embodiment includes a display unit 101 that displays an image, a plurality of light sources (first light source 103A, second light source 103B, third light source 103C, fourth light source 103D) that emit detection light and irradiate it onto at least one eyeball 111 of the subject, a stereo camera device 102 that captures an image of the eyeball 111 irradiated with the detection light, a position detection unit 24 that detects from the captured image the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, a gaze point detection unit 26 that calculates the position of the subject's gaze point based on the position of the pupil center and the position of the corneal reflex center, and a light source control unit 22 that switches the light source that emits detection light from among the plurality of light sources based on the target distance between the pupil center and the corneal reflex center.

[0087] In addition, the gaze detection method of this embodiment includes displaying an image on the display unit 101, emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject, capturing an image of the eyeball 111 irradiated with the detection light, detecting from the captured image the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature, and switching the light source that emits the detection light from among the multiple light sources based on the target distance between the pupil center and the corneal reflex center.

[0088] In addition, the gaze detection program of this embodiment causes a computer to perform the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature; and switching the light source that emits detection light from among the multiple light sources based on the target distance between the pupil center and the corneal reflex center.

[0089] According to the configuration of this embodiment, the light source that emits the detection light can be switched from among the multiple light sources based on the target distance between the pupil center and the corneal reflection center, which prevents the reflected image of the detection light irradiated onto the subject's eyeball from extending outside the cornea, thereby preventing a decrease in the accuracy of gaze detection.

[0090] In the gaze detection device 100 according to this embodiment, the multiple light sources include a first light source 103A and a second light source 103B installed below the display unit 101, and a third light source 103C and a fourth light source 103D installed above the display unit 101, the distance being a target distance DY in the vertical direction, and the light source control unit 22 switches the light source that emits the detection light between the lower light source and the upper light source. With this configuration, the light source that emits the detection light is switched between the lower light source and the upper light source based on the target distance DY in the vertical direction, so that a reflected image of the detection light can be reliably positioned within the cornea of the subject.

[0091] In the gaze detection device 100 according to this embodiment, the light source control unit 22 switches the light source that emits the detection light based on whether or not the shape of the corneal reflection image 113 of the detection light in the image of the eyeball 111 is included in the reference shape. This allows the light source that emits the detection light to be switched appropriately even when the positions of the pupil center and the corneal reflection center cannot be obtained.

[0092] In addition, the gaze detection device 100 of this embodiment includes a display unit 101 that displays an image, multiple light sources (first light source 103A, second light source 103B, third light source 103C, fourth light source 103D) that emit detection light and irradiate it onto at least one eyeball 111 of the subject, a stereo camera device 102 that captures an image of the eyeball 111 irradiated with the detection light, a position detection unit 24 that detects from the captured image the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, a gaze point detection unit 26 that calculates the position of the subject's gaze point based on the position of the pupil center and the position of the corneal reflex center, and a light source control unit 22 that switches the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflex image 113 of the detection light in the image of the eyeball 111 is included in a reference shape.

[0093] In addition, the gaze detection method of this embodiment includes displaying an image on the display unit 101, emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject, capturing an image of the eyeball 111 irradiated with the detection light, detecting from the captured image the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflection center indicating the center of the corneal reflection, calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature, and switching the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflection image 113 of the detection light in the image of the eyeball 111 is included in a reference shape.

[0094] In addition, the gaze detection program of this embodiment causes a computer to perform the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflection center indicating the center of the corneal reflection; calculating the position of the subject's point of gaze based on the position of the pupil center and the position of the center of corneal curvature; and switching the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflection image 113 of the detection light in the image of the eyeball 111 is included in a reference shape.

[0095] According to the configuration of this embodiment, the light source that emits the detection light can be switched from among the multiple light sources based on whether the shape of the corneal reflection image 113 of the detection light in the image of the eyeball 111 is included in the reference shape, thereby preventing the reflection image of the detection light irradiated onto the subject's eyeball from extending outside the cornea, thereby preventing a decrease in the accuracy of gaze detection.

[0096] In the gaze detection device 100 according to this embodiment, the reference shape is a single ellipse with a flattening ratio less than a threshold, and the light source control unit 22 switches the light source that emits the detection light when it is determined that the shape of the corneal reflection image 113 is not included in the reference shape. With this configuration, when it is determined that the shape of the corneal reflection image 113 is not included in the range of a single ellipse with a flattening ratio less than the threshold, the light source that emits the detection light is switched, so that the corneal reflection image 113 of the detection light can be positioned within the cornea of the subject.

[0097] In the gaze detection device 100 according to this embodiment, the multiple light sources include a first light source 103A and a second light source 103B installed below the display unit 101, and a third light source 103C and a fourth light source 103D installed above the display unit 101, and the light source control unit 22 switches the light source that emits the detection light between the lower light source and the upper light source. With this configuration, it is possible to reliably position the reflected image of the detection light within the cornea of the subject.

[0098] Although the embodiments of the present invention have been described above, the embodiments are not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0099] For example, in the above embodiment, a configuration has been described in which the plurality of light sources are lower light sources (first light source 103A, second light source 103B) arranged at the bottom of display unit 101 and upper light sources (third light source 103C, fourth light source 103D) arranged at the top of display unit 101. However, the present invention is not limited to this. For example, in addition to the lower light sources and upper light sources, or instead of at least one of these light sources, light sources may be arranged on the left and right sides of display unit 101.

[0100] In the above embodiment, the configuration in which two light sources (third light source 103C and fourth light source 103D) are provided as the upper light source has been described as an example, but the present invention is not limited to this. For example, the configuration in which one light source is provided as the upper light source may also be used.

[0101] In the above embodiment, the first camera 102A and the second camera 102B are disposed below the display unit 101. However, the present invention is not limited to this. The first camera 102A and the second camera 102B may be disposed above or to the side of the display unit 101.

[0102] In the above embodiment, the light source control unit 22 emits detection light from one of the lower light sources (first light source 103A, second light source 103B) and the upper light source (third light source 103C, fourth light source 103D). Alternatively, the light source control unit 22 may emit detection light from both the lower light source and the upper light source in sequence, and calculate the distance between the pupil center and the corneal reflex center in the image data of the eyeball 111. In this case, the gaze point detection unit 26 may detect the gaze point using image data obtained when detection light is emitted from either the lower light source or the upper light source, whichever light source produces the shorter calculated distance. Alternatively, the light source control unit 22 may emit detection light from both the lower light source and the upper light source in sequence, and calculate the shape of the corneal reflex image in the image data of the eyeball 111. In this case, the gaze point detection unit 26 may detect the gaze point using image data obtained when detection light is emitted from either the lower light source or the upper light source, whichever light source produces the closest calculated shape of the corneal reflex image.

[0103] This disclosure includes matters that contribute to the realization of the SDG "Good health and well-being for all" and contribute to value creation through healthcare products and services. [Explanation of symbols]

[0104] DX, DY... object distance, Q... reference shape, Ra, Rb, 127... distance, 20... computer system, 20A... calculation processing device, 20B... storage device, 20C... computer program, 21... display control unit, 22... light source control unit, 23... image data acquisition unit, 24... position detection unit, 25... center of curvature calculation unit, 26... gaze point detection unit, 27... storage unit, 28... input / output control unit, 30... input / output interface device, 40... drive circuit, 50... output device, 60... input device, 100... gaze detection device, 101... display unit, 101S... display screen, 102... stereo camera device, 102A... first camera, 102B... second camera, 10 2C, 103C... third light source, 103... illumination device, 103A... first light source, 103B... second light source, 103D... fourth light source, 103V... virtual light source, 109... corneal curvature radius, 110... corneal curvature center, 111... eyeball, 111a... cornea, 111b... sclera, 112... pupil, 112C... pupil center, 113... corneal reflection image, 113C, 121, 122, 124... corneal reflection center, 123, 173, 178... straight line, 130... target position, 141... first straight line, 142... second straight line, 166... intersection, 302... input / output unit, 402... display device driver, 404A... first camera input / output unit, 404B... second camera input / output unit, 406... light source driver

Claims

1. a display unit for displaying an image; a plurality of light sources that emit detection light and irradiate it onto at least one eyeball of the subject; an imaging unit that captures an image of the eyeball irradiated with the detection light; a position detection unit that detects, from the captured image, the position of a pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of a corneal reflection center indicating the center of the corneal reflection; a gaze point detection unit that calculates the position of the gaze point of the subject based on the position of the pupil center and the position of the corneal curvature center of the eyeball irradiated with the detection light; a light source control unit that compares a shape of a reflected image of the detection light in the image of the eyeball with a reference shape, and switches the light source that emits the detection light among the plurality of light sources based on whether the shape of the reflected image is included in the reference shape or not based on a matching rate; A gaze detection device comprising:

2. the reference shape is an elliptical shape having a flattening ratio less than a threshold value, When it is determined that the shape of the reflected image is not included in the reference shape, the light source control unit switches the light source that emits the detection light. The gaze detection device according to claim 1 .

3. the plurality of light sources include a lower light source disposed below the display unit and an upper light source disposed above the display unit; The light source control unit switches the light source that emits the detection light between the lower light source and the upper light source.

3. The gaze detection device according to claim 1.

4. Displaying an image on a display unit; emitting detection light from a plurality of light sources and irradiating the light onto at least one eyeball of the subject; capturing an image of the eyeball illuminated with the detection light; From the captured image, a pupil center indicating the center of the pupil of the eyeball irradiated with the detection light is obtained. detecting a position and a position of a corneal reflex center indicating a center of the corneal reflex; calculating a position of the gaze point of the subject based on the position of the pupil center and the position of the corneal curvature center of the eyeball irradiated with the detection light; comparing a shape of a reflected image of the detection light in the image of the eyeball with a reference shape, and switching the light source that emits the detection light from among the plurality of light sources based on whether or not the shape of the reflected image is included in the reference shape based on a rate of match; A gaze detection method including:

5. A process of displaying an image on a display unit; a process of emitting detection light from a plurality of light sources and irradiating it onto at least one eyeball of the subject; capturing an image of the eyeball illuminated with the detection light; From the captured image, a pupil center indicating the center of the pupil of the eyeball irradiated with the detection light is obtained. a process of detecting the position of the corneal reflection center indicating the center of the corneal reflection; a process of calculating the position of the gaze point of the subject based on the position of the pupil center and the position of the corneal curvature center of the eyeball irradiated with the detection light; a process of comparing a shape of a reflected image of the detection light in the image of the eyeball with a reference shape, and switching the light source that emits the detection light among the plurality of light sources based on whether or not the shape of the reflected image is included in the reference shape based on a matching rate; A gaze detection program that causes a computer to execute the above.

Images