Pupillary detection device and pupilary detection method

Abstract

This stabilizes the detection accuracy when continuously detecting the positions of the left and right pupils of a subject over time. [Solution] The pupil detection device 1 comprises a camera 10 that acquires an eye image of subject A, a light source 13 provided on the camera 10, a lighting control unit 21 that controls the light intensity of the light source 13, and a calculation unit 23. The calculation unit 23 detects the right pupil based on the eye image of the first frame, the left pupil based on the eye image of the second frame following the first frame, the right pupil based on the eye image of the third frame following the first frame, and the left pupil based on the eye image of the fourth frame following the second frame. The lighting control unit 21 adjusts the light intensity at the timing of the third frame based on the brightness of the eye area corresponding to the right pupil detected from the eye image of the first frame, and adjusts the light intensity at the timing of the fourth frame based on the brightness of the eye area corresponding to the left pupil detected from the eye image of the second frame.

Description

[Technical Field]

[0001] The present invention relates to a pupil detection device and a pupil detection method for detecting pupils from images of people. [Background technology]

[0002] In recent years, devices that detect the position of a person's pupil from images obtained using a light source such as a near-infrared light source and a video camera have become widespread for purposes such as gaze detection (see Patent Document 1 and Patent Document 2 below). These devices acquire an image (bright pupil image) by shining light that tends to make the pupil relatively brighter onto the subject's face, and an image (dark pupil image) by shining light that tends to make the pupil relatively darker onto the subject's face. Subsequently, the position of the subject's pupil is detected by calculating a difference image using these images. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2008-29702 [Patent Document 2] Japanese Patent Publication No. 2018-28728 [Overview of the project] [Problems that the invention aims to solve]

[0004] When detecting the positions of a subject's left and right pupils using the device described above, the detection accuracy is greatly affected by the relative positions of the subject's left and right pupils within the video camera's field of view. For example, due to the characteristics of the video camera's lens, the center of the image is generally brighter and the edges are darker. Therefore, the detection accuracy is relatively high when either of the left or right pupils is located in the center of the field of view, but tends to be relatively low when either pupil is located at the edge of the field of view. In addition, the detection accuracy of each pupil tends to be affected by its position within the field of view due to the directionality of the light source.

[0005] This invention has been made in view of the above problems, and aims to provide a pupil detection device that can stabilize the detection accuracy when continuously detecting the positions of the left and right pupils of a subject over time. [Means for solving the problem]

[0006] To solve the above problems, a pupil detection device according to one embodiment of the present invention comprises a camera that acquires eye images in a series of frames by imaging the eyes of a subject, a light source provided outside the aperture of the camera that illuminates the pupils of the subject, a light source control unit that controls the amount of light from the light source, and a calculation unit that detects the positions of the left and right pupils of the subject based on the eye images, wherein the calculation unit detects at least one of the left and right pupils based on the eye image of the first frame, and based on the eye image of the second frame following the first frame, it detects the positions of the left and right pupils The light source control unit detects at least the pupil of the other eye, at least the pupil of the other eye based on the eye image of the third frame after the first frame, at least the pupil of the other eye based on the eye image of the fourth frame after the second frame, and adjusts the light intensity at the timing of acquiring the eye image of the third frame based on the brightness of the eye part corresponding to the pupil of the other eye detected from the eye image of the first frame, and adjusts the light intensity at the timing of acquiring the eye image of the fourth frame based on the brightness of the eye part corresponding to the other pupil detected from the eye image of the second frame.

[0007] Alternatively, another embodiment of the present invention relates to a pupil detection method using a camera that acquires eye images in a series of frames by imaging the eyes of a subject, a light source provided outside the aperture of the camera to illuminate the pupils of the subject, control of the camera and the light source, and an image processing device that performs image processing to detect the positions of the left and right pupils of the subject, wherein at least one of the left and right pupils is detected based on the eye image of the first frame, and the positions of the left and right pupils are detected based on the eye image of the second frame following the first frame. At least one pupil of the eye is detected, at least one pupil of the eye is detected based on the eye image of the third frame after the first frame, at least one pupil of the eye is detected based on the eye image of the fourth frame after the second frame, the amount of light from the light source at the time of acquiring the eye image of the third frame is adjusted based on the brightness of the eye part corresponding to one pupil detected from the eye image of the first frame, and the amount of light at the time of acquiring the eye image of the fourth frame is adjusted based on the brightness of the eye part corresponding to the other pupil detected from the eye image of the second frame.

[0008] According to the pupil detection device or pupil detection method described above, the pupils of the subject are illuminated by a light source placed outside the camera's aperture, and at the same time, the camera acquires eye images in a series of frames. The calculation unit continuously detects the position of one of the pupils, the left or the right, based on the eye image of the first frame and the eye image of the third frame thereafter, and continuously detects the position of the other pupil, the left or the right, based on the eye image of the second frame following the first frame and the eye image of the fourth frame thereafter. When acquiring these eye images, the light source control unit adjusts the amount of light from the light source at the time of acquiring the eye image of the third frame based on the brightness of the eye area corresponding to one pupil detected from the eye image of the first frame, and adjusts the amount of light from the light source at the time of acquiring the eye image of the fourth frame based on the brightness of the eye area corresponding to the other pupil detected from the eye image of the second frame. This allows for stabilization of the brightness of the portion corresponding to each pupil of the target in the eye image of each frame, enabling stable detection of the position of each pupil using eye images from separate frames. As a result, the detection accuracy can be stabilized when detecting the positions of the left and right pupils of a subject in a continuous time sequence.

[0009] Here, the light source includes a first light source provided outside the camera aperture to illuminate the subject's pupil relatively brightly to the camera, and a second light source provided outside the camera aperture to illuminate the subject's pupil relatively darkly to the camera. The calculation unit detects at least one pupil based on a first calculated image obtained by calculating the eye image of the first bright pupil acquisition frame and the eye image of the first dark pupil acquisition frame, and then calculates the second bright pupil acquisition after the first bright pupil acquisition frame. Based on a second calculated image obtained by combining the eye image of the acquired frame and the eye image of the second dark pupil acquisition frame after the first dark pupil acquisition frame, at least the other pupil is detected. Based on a third calculated image obtained by combining the eye image of the third bright pupil acquisition frame after the first bright pupil acquisition frame and the eye image of the third dark pupil acquisition frame after the first dark pupil acquisition frame, at least one pupil is detected. The light source control unit may detect at least the other pupil based on a fourth calculated image obtained by calculating the eye image of one pupil and the eye image of the fourth dark pupil acquisition frame after the third dark pupil acquisition frame, and the light source control unit may turn on the first light source in the first bright pupil acquisition frame, the second bright pupil acquisition frame, the third bright pupil acquisition frame, and the fourth bright pupil acquisition frame, and turn on the second light source in the first dark pupil acquisition frame, the second dark pupil acquisition frame, the third dark pupil acquisition frame, and the fourth dark pupil acquisition frame, and in the third bright pupil acquisition frame or the third dark pupil acquisition frame, adjust the light intensity of the first or second light source based on the brightness of the eye part corresponding to one pupil detected from the first calculated image, and in the fourth bright pupil acquisition frame or the fourth dark pupil acquisition frame, adjust the light intensity of the first or second light source based on the brightness of the eye part corresponding to the other pupil detected from the second calculated image.

[0010] In this case, the subject's pupils are intermittently illuminated by a first and second light source positioned outside the camera's aperture, and at the same time, the camera acquires eye images in a series of frames. As a result, eye images for the first to fourth bright pupil acquisition frames and eye images for the first to fourth dark pupil acquisition frames are acquired. Based on the first and third calculated images, the calculation unit continuously detects the position of one of the left or right pupils, and based on the second and fourth calculated images, it continuously detects the position of the other of the left or right pupils. When acquiring these eye images, the light source control unit adjusts the light intensity of the light source at the time of acquiring the eye image in the third bright pupil acquisition frame or the third dark pupil acquisition frame, based on the brightness of the eye area corresponding to one pupil detected from the first calculated image. Similarly, the light intensity of the light source at the time of acquiring the eye image in the fourth bright pupil acquisition frame or the fourth dark pupil acquisition frame is adjusted based on the brightness of the eye area corresponding to the other pupil detected from the second calculated image. This stabilizes the brightness of the parts corresponding to each pupil of the target in each calculated image, allowing for stable detection of the position of each pupil using separate calculated images. As a result, the detection accuracy can be stabilized when detecting the positions of the left and right pupils of a subject in a continuous time sequence.

[0011] Furthermore, the light source is provided outside the camera's aperture and simultaneously emits a first light component that makes the subject's pupil appear relatively bright to the camera and a second light component that makes the subject's pupil appear relatively dark to the camera. The calculation unit obtains a first calculated image by calculating a bright pupil image in which one pupil appears relatively bright and a dark pupil image in which the other pupil appears relatively dark, based on the eye image of the first frame, and detects at least one pupil based on the first calculated image. Based on the eye image of the second frame, it obtains a second calculated image by calculating a bright pupil image in which the other pupil appears relatively bright and a dark pupil image in which the other pupil appears relatively dark, based on the eye image of the second frame, and detects at least the other pupil based on the second calculated image. Based on the eye image of the third frame, it detects one pupil A third calculated image is obtained by calculating the bright pupil image of one pupil and the dark pupil image of the other pupil, and at least one pupil is detected based on the third calculated image. A fourth calculated image is obtained by calculating the bright pupil image of the other pupil and the dark pupil image of the other pupil based on the eye image of the fourth frame, and at least the other pupil is detected based on the fourth calculated image. The light source control unit may adjust the light intensity at the timing of obtaining the eye image of the third frame based on the brightness of the eye area corresponding to one pupil in the first calculated image obtained from the eye image of the first frame, and adjust the light intensity at the timing of obtaining the eye image of the fourth frame based on the brightness of the eye area corresponding to the other pupil in the second calculated image obtained from the eye image of the second frame.

[0012] In this case, the subject's pupils are illuminated by a first and second light component emitted from a light source positioned outside the camera's aperture, and at the same time, the camera acquires eye images in a series of frames, thereby acquiring eye images for the first to fourth frames. The calculation unit then continuously detects the position of one of the pupils (left or right) based on the first and third calculated images, and continuously detects the position of the other pupil (left or right) based on the second and fourth calculated images. When acquiring these eye images, the light source control unit adjusts the amount of light from the light source at the time of acquiring the third frame's eye image based on the brightness of the eye area corresponding to one pupil detected from the first calculated image, and adjusts the amount of light from the light source at the time of acquiring the fourth frame's eye image based on the brightness of the eye area corresponding to the other pupil detected from the second calculated image. This stabilizes the brightness of the parts corresponding to each pupil of the detection target in each calculated image, allowing for stable detection of the position of each pupil using separate calculated images. As a result, the detection accuracy can be stabilized when detecting the positions of the left and right pupils of a subject over time.

[0013] Furthermore, the calculation unit may acquire the first to fourth calculated images as difference images. In this case, the positions of the subject's left and right pupils can be stably detected without being affected by the surrounding environment.

[0014] Furthermore, the light source control unit may adjust the light intensity based on the brightness of one or the other pupil in the difference image. This allows the brightness of the pupil to be detected in the difference image to be adjusted to a desired brightness, thereby further stabilizing the detection accuracy of the positions of the left and right pupils.

[0015] Furthermore, the light source control unit may adjust the light intensity by changing the emission time width. In this case, the detection accuracy of the positions of the left and right pupils can be stabilized by simple adjustments.

[0016] Further, the light source may be provided outside the opening of the camera. In this case, the pupil of the subject can be accurately detected.

Advantages of the Invention

[0017] According to the present disclosure, it is possible to stabilize the detection accuracy when detecting the positions of the left and right pupils of a subject continuously in time.

Brief Description of the Drawings

[0018] [Figure 1] It is a perspective view showing a pupil detection device according to an embodiment. [Figure 2] It is a plan view showing a lens portion of a camera. [Figure 3] It is a diagram showing a hardware configuration of an image processing device according to an embodiment. [Figure 4] It is a block diagram showing a functional configuration of a pupil detection device according to an embodiment. [Figure 5] It is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 in FIG. 4. [Figure 6] It is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 in FIG. 4. [Figure 7] It is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 in FIG. 4. [Figure 8] It is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 according to the first modification. [Figure 9] It is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 according to the second modification. [Figure 10] It is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 according to the third modification. [Figure 11] It is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 according to the third modification. [Modes for carrying out the invention]

[0019] Hereinafter, preferred embodiments of the pupil detection device and pupil detection method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0020] First, the configuration of the pupil detection device 1 according to the embodiment of this disclosure will be explained using Figures 1 to 4. The pupil detection device 1 is a computer system that detects the positions of the left and right pupils of subject A by imaging the eyes of subject A. A subject is a person whose pupil position is to be detected, and can also be called a test subject. The purpose of use of the pupil detection device 1 is not limited in any way, and for example, the pupil detection device 1 can be used for detecting distracted driving, confirming the driver's safety checks of the side mirrors and rearview mirror, detecting driver drowsiness, investigating the degree of interest in products, inputting data into computers used in amusement devices, and as a diagnostic device for diagnosing autism in infants and young children.

[0021] As schematically shown in Figure 1, the pupil detection device 1 comprises a pair of cameras (a first camera and a second camera) 10 that function as a stereo camera and an image processing device 20. In the following, the pair of cameras 10 will be described as the left camera 10 located to the left of subject A, as needed. L And, the right camera 10 located to the right of subject A. R A distinction is made between the two. In this embodiment, the pupil detection device 1 further includes a display device 30 which is the object that subject A looks at, but the purpose of use of the pupil detection device 1 is not limited as described above, so the object in the line of sight of subject A is not limited to the display device 30, but could be, for example, the windshield of a car. Therefore, the display device 30 is not an essential element of the pupil detection device 1. Each camera 10 is connected to the image processing device 20 by wireless or wired connection, and various data or commands are sent and received between the camera 10 and the image processing device 20. Camera calibration is performed in advance for each camera 10.

[0022] Camera 10 is used to photograph the eyes of subject A, including the left and right pupils and their surrounding areas. The pair of cameras 10 are arranged horizontally at a predetermined distance apart and are positioned lower than subject A's face to prevent reflections from appearing in the facial image when subject A is wearing glasses. The elevation angle of the cameras 10 relative to the horizontal is set to a range of, for example, 20 to 35 degrees, taking into consideration both reliable detection of the pupils and avoiding obstruction of subject A's field of view. Camera calibration is performed in advance for each camera 10.

[0023] In this embodiment, each camera 10 is an imaging device capable of capturing images in temporally continuous frames at predetermined time intervals (for example, 120 fps intervals). In response to a command from the image processing device 20, the camera 10 captures images of subject A, acquires eye image data showing the left and right pupils and the two eye areas including them, and outputs this data to the image processing device 20.

[0024] Figure 2 schematically shows the lens portion of camera 10. As shown in this figure, in camera 10, the objective lens 11 is housed in a circular opening 12, and a light source 13 is mounted outside the opening 12. The light source 13 is a device for irradiating illumination light towards the eyes of subject A, and consists of a plurality of light-emitting elements 13a (first light source) and a plurality of light-emitting elements 13b (second light source). The light-emitting elements 13a are semiconductor light-emitting elements (LEDs) with a central wavelength of output light of 850 nm, and are arranged in a ring shape at equal intervals along the edge of the opening 12. The light-emitting elements 13b are semiconductor light-emitting elements with a central wavelength of output light of 940 nm, and are arranged vertically in a vertical direction to the left and right outside the light-emitting elements 13a. The distance from the optical axis of camera 10 to the light-emitting elements 13b is greater than the distance from the optical axis to the light-emitting elements 13a. Each light-emitting element 13a, 13b is provided to emit illumination light along the optical axis of camera 10. Note that the arrangement of the light source 13 is not limited to the configuration shown in Figure 2; other arrangements are also acceptable as long as the camera 10 can be considered as a pinhole model. The light-emitting elements 13a and 13b each emit illumination light at different timings and for different durations in response to commands from the image processing device 20. The intensity of the illumination light emitted by the light-emitting elements 13a and 13b is pre-set so that when illuminated on subject A, the background other than the pupil is the same brightness.

[0025] The light-emitting element (first light source) 13a constituting the light source 13 is an illumination light (first illumination light, first light component) for obtaining a bright pupil image, and is a light source that illuminates the face of subject A with light that makes the pupil of subject A appear relatively bright to the camera 10. A bright pupil image is an image in which the pupil of subject A appears relatively brighter compared to the dark pupil image described later. The light-emitting element 13a is also used as a light source to generate spectacle reflection when acquiring a dark pupil image and cancel out the spectacle reflection in the difference image.

[0026] The light-emitting element (second light source) 13b constituting the light source 13 is an illumination light (second illumination light, second light component) for obtaining a dark pupil image, and is a light source that illuminates subject A's face with light that makes subject A's pupil appear relatively dark to the camera 10. A dark pupil image is an image in which subject A's pupil appears relatively darker compared to the aforementioned bright pupil image. This light-emitting element 13b is also used as a light source to enhance and produce corneal reflection when acquiring both the bright pupil image and the dark pupil image.

[0027] The image processing device 20 is a computer that controls the camera 10 and the light source 13, and detects the pupil position and gaze direction of subject A. The image processing device 20 may be constructed using a stationary or portable personal computer (PC), a workstation, or other types of computers. Alternatively, the image processing device 20 may be constructed using a combination of multiple computers of any type. When multiple computers are used, these computers are connected via a communication network such as the Internet or an intranet.

[0028] Figure 3 shows a typical hardware configuration of the image processing device 20. The image processing device 20 includes a CPU (processor) 101 that runs an operating system and application programs, a main memory unit 102 consisting of ROM and RAM, an auxiliary memory unit 103 consisting of a hard disk or flash memory, a communication control unit 104 consisting of a network card or wireless communication module, an input device 105 such as a keyboard or mouse, and an output device 106 such as a display or printer.

[0029] Each functional element of the image processing device 20, described later, is realized by loading predetermined software onto the CPU 101 or main memory unit 102, operating the communication control unit 104, input device 105, output device 106, etc., under the control of the CPU 101, and reading and writing data to the main memory unit 102 or auxiliary memory unit 103. The data and database necessary for processing are stored in the main memory unit 102 or auxiliary memory unit 103.

[0030] As shown in Figure 4, the image processing device 20 comprises a lighting control unit (light source control unit) 21, an image acquisition unit 22, and a calculation unit 23 as functional components. The lighting control unit 21 controls the lighting timing and duration of the light source 13. The image acquisition unit 22 is a functional element that acquires image data (eye image data) from the camera 10 by controlling the shooting timing of the camera 10 in synchronization with the lighting timing of the light source 13. The calculation unit 23 is a functional element that detects the three-dimensional position of the pupil of subject A and the gaze vector as the direction of subject A's gaze based on the image data. A gaze is a line connecting the center of subject A's pupil and the point of fixation of subject A (the point the subject is looking at). The term "gaze" includes the meaning (concept) of a starting point, an ending point, and a direction. A "gaze vector" is a vector representation of the direction of the subject's gaze and is one form of representing the "direction of gaze". The output destination of the pupil position and gaze direction detected by the image processing device 20 is not limited in any way. For example, the image processing device 20 may display the determination result on a monitor as an image, graphic, or text, store it in a storage device such as memory or a database, or transmit it to another computer system via a communication network.

[0031] The image acquisition unit 22 is a functional element that controls the shooting timing of the camera 10. Specifically, the image acquisition unit 22 controls the camera 10 to repeatedly capture images at a predetermined frame rate (e.g., 120 fps) and a predetermined exposure time, and to repeatedly acquire a right bright pupil image, a right dark pupil image, a dark pupil image, for detecting the position of the right pupil, a left bright pupil image, a bright pupil image, for detecting the position of the left pupil, and a left dark pupil image, for detecting the position of the left pupil.

[0032] Figure 5 is a conceptual diagram showing the temporal relationship of consecutive frames of image data acquired by the image acquisition unit 22. Thus, when imaging by the camera 10 begins, the following frames are set in this order: the right bright pupil acquisition frame (first bright pupil acquisition frame), which is the frame for acquiring the right bright pupil image; the right dark pupil acquisition frame (first dark pupil acquisition frame), which is the frame for acquiring the right dark pupil image; the left bright pupil acquisition frame (second bright pupil acquisition frame), which is the frame for acquiring the left bright pupil image; and the left dark pupil acquisition frame (second dark pupil acquisition frame), which is the frame for acquiring the left dark pupil image. Similarly, immediately after the left dark pupil acquisition frame, the right bright pupil acquisition frame (third bright pupil acquisition frame), the right dark pupil acquisition frame (third dark pupil acquisition frame), the left bright pupil acquisition frame (fourth bright pupil acquisition frame), and the left dark pupil acquisition frame (fourth dark pupil acquisition frame) are set in this order, and thereafter the 8 frames from the right bright pupil acquisition frame to the left dark pupil acquisition frame are repeated. The second bright pupil acquisition frame is set to follow the first bright pupil acquisition frame, the second dark pupil acquisition frame is set to follow the first dark pupil acquisition frame, the third bright pupil acquisition frame is set to follow the first bright pupil acquisition frame, and the third dark pupil acquisition frame is set to follow the first dark pupil acquisition frame. Also, the fourth bright pupil acquisition frame is set to follow the third bright pupil acquisition frame, and the fourth dark pupil acquisition frame is set to follow the third dark pupil acquisition frame.

[0033] The lighting control unit 21 is a functional element that controls the lighting timing of the light-emitting elements 13a and 13b and the amount of light emitted (light intensity) during the exposure period of the camera 10, in synchronization with the shooting timing of the camera 10. In this embodiment, the lighting control unit 21 controls the amount of light emitted by the light-emitting elements 13a and 13b by setting their respective lighting periods (light emission time widths). Specifically, the lighting control unit 21 lights up the light-emitting elements 13a and 13b in such a way that it repeatedly lights up the light-emitting element 13a relatively strongly (for a relatively long lighting period) in synchronization with the first to fourth bright pupil acquisition frames, which are the imaging timings for bright pupil images, and lights up the light-emitting element 13b relatively strongly (for a relatively long lighting period) in synchronization with the first to fourth dark pupil acquisition frames, which are the imaging timings for dark pupil images. At this time, the lighting control unit 21 controls the light-emitting element 13b to light up at a relatively low intensity (for a relatively short lighting period) in order to enhance the corneal reflection image when capturing a bright pupil image, and controls the light-emitting element 13a to light up at a relatively low intensity (for a relatively short lighting period) when capturing a dark pupil image in order to generate spectacle reflection and cancel out the spectacle reflection in the difference image described later.

[0034] Furthermore, the lighting control unit 21 has the function of adjusting (controlling) the light intensity of the light-emitting element 13a at the time of capturing a bright pupil image and the light intensity of the light-emitting element 13b at the time of capturing a dark pupil image, according to the detection result by the calculation unit 23, which will be described later (details will be described later). The lighting control unit 21 adjusts the light intensity for one camera 10 included in a pair of cameras 10. L and one camera 10 R Each has a function to be executed. In this case, camera 10 L The timing of shooting and lighting in the camera 10 R The timing of the photo being taken and the timing of the lights being turned on are set to be staggered within the duration of one frame.

[0035] Due to the functions of the image acquisition unit 22 and the lighting control unit 21, a bright pupil image is acquired when the pupil is captured by the camera 10 at the timing when illumination light is emitted from the light-emitting element 13a to the eyeball of subject A, and a dark pupil image is acquired when the pupil is captured by the camera 10 at the timing when illumination light is emitted from the light-emitting element 13b. This is due to the following properties: In other words, when the illumination light to the eyeball of subject A is incident from a position relatively far from the optical axis of the camera 10, the illumination light that enters through the pupil of the eyeball, is reflected inside the eyeball and passes through the pupil again does not easily reach the camera 10, so the pupil appears relatively dark. Conversely, when the illumination light to the eyeball of subject A is incident from a position relatively close to the optical axis of the camera 10, the illumination light that enters through the pupil of the eyeball, is reflected inside the eyeball and passes through the pupil again easily reaches the camera 10, so the pupil appears relatively bright.

[0036] The following describes in detail the functions of the calculation unit 23 for detecting the three-dimensional position of subject A's pupil and the direction of subject A's gaze. These detection functions are performed independently for subject A's right and left pupils.

[0037] The calculation unit 23 uses two images, a bright pupil image and a dark pupil image, acquired in different frames, to obtain a series of difference images by subtracting the brightness of each pixel in the dark pupil image window from the brightness of each pixel in the bright pupil image window. At this time, considering that there is a time difference between the acquisition of the bright pupil image and the dark pupil image, and that subject A moves relative to the camera 10, the calculation unit 23 uses a method similar to that described in Japanese Patent No. 4452836 to detect the position of the corneal reflection image on the bright pupil image and the dark pupil image, and corrects the position of the window on the dark pupil image relative to the bright pupil image so that the position of the corneal reflection image matches. The position of the corneal reflection image on the image is detected by binarizing it with a corneal reflection threshold higher than the pupil brightness obtained from the previous frame, performing a labeling process, selecting the pixel group that has the size and shape best as a corneal reflection, and then determining the center of the corneal reflection image from that pixel group as the centroid, taking brightness into consideration. On the other hand, the calculation unit 23 may detect the position of the corneal reflection image on the bright pupil image and the dark pupil image, and correct the position of the window on the bright pupil image relative to the dark pupil image so that the positions of the corneal reflection images match. With this function, the calculation unit 23 can continuously acquire difference images at predetermined time intervals.

[0038] The calculation unit 23 then uses the consecutively acquired difference images to detect the position of the pupil image of subject A on the image (hereinafter also referred to as the two-dimensional position). Specifically, the calculation unit 23 binarizes the difference image based on a pre-set pupil threshold, performs isolated point removal, noise reduction by morphological processing, and labeling. The calculation unit 23 then detects the group of pixels that has the most pupil-like shape as the pupil. At this time, even if the pupil is hidden by the eyelid or eyelashes, the boundary between the eyelid or eyelashes and the pupil is excluded as a false pupil contour, and only the true pupil contour is fitted with an ellipse to detect the position of the true pupil contour on the multiplied image, and the center position of the pupil image is calculated from the ellipse equation obtained by ellipse fitting. At this time, in order to reduce the number of bits of the pixel value of the multiplied image to be calculated, the calculation unit 23 may divide the pixel value of the multiplied image by a predetermined value (for example, 2 to the power of 8 if the pixel value is represented by 8 bits) before detecting the two-dimensional position of the pupil image.

[0039] Furthermore, the calculation unit 23 continuously acquires, for each pair of cameras 10, the two-dimensional position of the pupil image detected based on the difference image obtained from the image data of two identical frames, and the position of the corneal reflection image detected on the dark pupil image or bright pupil image that was the source of the difference image. Then, the calculation unit 23 calculates the three-dimensional position of the pupil of subject A using the two-dimensional position of the pupil image obtained corresponding to the pair of cameras 10. Furthermore, the calculation unit 23 continuously detects the gaze vector and gaze point of subject A at predetermined time intervals using the calculated three-dimensional position of the pupil, the two-dimensional position of the pupil image obtained for each pair of cameras 10, and the position of the corneal reflection image. The above methods for calculating the three-dimensional position of the pupil, the gaze vector, and the gaze point can employ methods developed by the inventors (see International Publication WO2012 / 020760).

[0040] In the frame setting example shown in Figure 5, the calculation unit 23 at least detects the two-dimensional position of subject A's right pupil based on a first difference image (calculated image) obtained by calculating the image data of the first bright pupil acquisition frame and the image data of the first dark pupil acquisition frame, and at least detects the two-dimensional position of subject A's left pupil based on a second difference image (calculated image) obtained by calculating the image data of the second bright pupil acquisition frame and the image data of the second dark pupil acquisition frame. Next, it at least detects the two-dimensional position of subject A's right pupil based on a third difference image (calculated image) obtained by calculating the image data of the third bright pupil acquisition frame and the image data of the third dark pupil acquisition frame, and at least detects the two-dimensional position of subject A's left pupil based on a fourth difference image (calculated image) obtained by calculating the image data of the fourth bright pupil acquisition frame and the image data of the fourth dark pupil acquisition frame.

[0041] The light intensity adjustment function of the lighting control unit 21, which utilizes the detection results from the calculation unit 23, is as follows. Specifically, the lighting control unit 21 adjusts the light intensity of the light-emitting elements 13a and 13b at the timing of acquiring image data of a later frame that detects the same pupil, based on the brightness of the eye area corresponding to the pupil to be detected, as detected from the image data of the previous frame. In the frame setting example shown in Figure 5, the lighting control unit 21 adjusts at least one (preferably both) of the lighting period of the light-emitting element 13a in the third bright pupil acquisition frame and the lighting period of the light-emitting element 13b in the third dark pupil acquisition frame, based on the brightness of the eye area corresponding to the right pupil detected from the first difference image. Furthermore, the lighting control unit 21 adjusts at least one (preferably both) of the lighting period of the light-emitting element 13a in the fourth bright pupil acquisition frame and the lighting period of the light-emitting element 13b in the fourth dark pupil acquisition frame, based on the brightness of the eye area corresponding to the left pupil detected from the second difference image. This adjustment can be performed using methods developed by the present inventors (see Japanese Patent Publication No. 6687195 or Japanese Patent Publication No. 7228885), and can be controlled using the luminance value of the pupil image in the difference image, control based on the luminance value of the corneal reflection image in the light pupil image and dark pupil image that formed the basis of the difference image, or control based on the luminance value of the iris around the pupil in the light pupil image and dark pupil image that formed the basis of the difference image.

[0042] In this embodiment, the image processing device 20 may set a time-series of frames as shown in Figures 6 and 7. In the example shown in Figures 6 and 7, the frames are set in the following order: right bright pupil acquisition frame (first bright pupil acquisition frame), left bright pupil acquisition frame (second bright pupil acquisition frame), right dark pupil acquisition frame (first dark pupil acquisition frame), left dark pupil acquisition frame (second dark pupil acquisition frame), right bright pupil acquisition frame (third bright pupil acquisition frame), left bright pupil acquisition frame (fourth bright pupil acquisition frame), right dark pupil acquisition frame (third dark pupil acquisition frame), and left dark pupil acquisition frame (fourth dark pupil acquisition frame). In other words, bright pupil images or dark pupil images are acquired alternately for the right pupil and the left pupil. In the example shown in Figure 6, the illumination period of the light-emitting elements 13a and 13b is controlled based on the luminance detected from the first difference image in the third bright pupil acquisition frame and the third dark pupil acquisition frame, and the illumination period of the light-emitting elements 13a and 13b is controlled based on the luminance detected from the second difference image in the fourth bright pupil acquisition frame and the fourth dark pupil acquisition frame. In the example shown in Figure 7, in the third bright pupil acquisition frame, the illumination period of the light-emitting element 13a is controlled based on the luminance detected from the first difference image; in the third dark pupil acquisition frame, the illumination period of the light-emitting element 13b is controlled based on the luminance detected from the difference image based on the image of the first dark pupil acquisition frame and the image of the third bright pupil acquisition frame; in the fourth bright pupil acquisition frame, the illumination period of the light-emitting element 13a is controlled based on the luminance detected from the second difference image; and in the fourth dark pupil acquisition frame, the illumination period of the light-emitting element 13b is controlled based on the luminance detected from the difference image based on the image of the second dark pupil acquisition frame and the image of the fourth bright pupil acquisition frame.

[0043] The effects and functions of the pupil detection device 1 described above will now be explained.

[0044] According to the pupil detection device 1 and the pupil detection method using the pupil detection device 1, the pupils of subject A are intermittently illuminated by light-emitting elements 13a and 13b positioned outside the aperture 12 of the camera 10, and at the same time, the camera 10 acquires eye images in a series of frames. As a result, eye images of the first to fourth bright pupil acquisition frames and eye images of the first to fourth dark pupil acquisition frames are acquired. The calculation unit 23 then continuously detects the position of the right pupil based on the first difference image and the third difference image, and continuously detects the position of the left pupil based on the second difference image and the fourth difference image. When acquiring these eye images, the lighting control unit 21 adjusts the light intensity of the light-emitting elements 13a and 13b at the time of acquiring the eye image in the third bright pupil acquisition frame or the third dark pupil acquisition frame, based on the brightness of the eye area corresponding to the right pupil detected from the first difference image. Similarly, the light intensity of the light-emitting elements 13a and 13b at the time of acquiring the eye image in the fourth bright pupil acquisition frame or the fourth dark pupil acquisition frame, based on the brightness of the eye area corresponding to the left pupil detected from the second difference image. This stabilizes the brightness of the parts corresponding to each pupil of the target in each difference image, allowing for stable detection of the position of each pupil using separate calculated images. As a result, the detection accuracy can be stabilized when detecting the positions of the left and right pupils of a subject in a continuous time sequence.

[0045] To prevent a decrease in the detection accuracy of the left and right pupil positions due to the directionality of the light source, it is conceivable to use a light source with weak directionality. However, in that case, the overall illumination intensity of the light source will decrease. For example, the illumination intensity is approximately nine times greater when using an LED with a directionality of ±15 degrees compared to when using an LED with a directionality of ±45 degrees. Therefore, in order to achieve miniaturization of the light source size or to ensure illumination intensity, it is preferable to use a light source with a certain degree of directionality. Generally, when a directional light source is used, the brightness of the pupil changes in accordance with the change in the position of the pupil in the camera's field of view, which may reduce the detection accuracy of its position. However, with the pupil detection device 1 of this embodiment, the detection accuracy of the left and right pupil positions can be stabilized even when a directional light source is used.

[0046] Furthermore, when detecting a subject with aneuploidy (a condition where the size of the left and right pupils differs), a difference in brightness between the right and left pupils may occur in the image acquired by the camera. For example, in a subject with aneuploidy, the difference in pupil size between the left and right pupils may exceed 1 mm, resulting in a large difference in brightness between the left and right pupils in the image. In this case, to detect the positions of the left and right pupils using a single image acquired by the camera, the light intensity of the light source must be increased. Increasing the light intensity of the light source can cause the brightness of the pupil with higher brightness to saturate, making it difficult to detect corneal reflection (see Patent No. 4452836), which is necessary for calculating the difference image when the subject's head moves. Ideally, it is desirable to keep the brightness of the pupils constant in the image to stabilize pupil detection. According to this embodiment, the brightness of the left and right pupils can be stabilized across multiple frames of images, so the detection accuracy of the positions of the left and right pupils using the difference image can be maintained at a high level across multiple consecutive frames.

[0047] Furthermore, the calculation unit 23 detects the position of the pupils using the difference image. In this case, the positions of the left and right pupils of subject A can be stably detected without being affected by ambient light.

[0048] Furthermore, the lighting control unit 21 adjusts the light intensity of the light-emitting elements 13a and 13b based on the brightness of the right or left pupil in the difference image. This allows the brightness of the pupil to be detected in the difference image to be adjusted to a desired brightness, thereby further stabilizing the detection accuracy of the positions of the left and right pupils.

[0049] Furthermore, the lighting control unit 21 adjusts the light intensity of the light-emitting elements 13a and 13b by changing the lighting period. In this case, the detection accuracy of the positions of the left and right pupils can be stabilized by simple adjustments.

[0050] The present invention is not limited to the embodiments described above. The configurations of the above embodiments can be modified in various ways.

[0051] In the embodiment described above, the calculation unit 23 detected the pupil position based on a difference image calculated from a bright pupil image and a dark pupil image. However, as a first modification, the pupil position may be detected using the image data acquired from the camera 10 as is. Figure 8 is a conceptual diagram showing the temporal relationship of image data of consecutive frames acquired by the image acquisition unit 22 according to the first modification. In this way, the image acquisition unit 22 sequentially acquires right pupil image data of the right pupil acquisition frame (first frame) for detecting the right pupil, right pupil image data of the left pupil acquisition frame (second frame) for detecting the left pupil, right pupil image data of the right pupil acquisition frame (third frame), and right pupil image data of the left pupil acquisition frame (fourth frame). In this first modification, the light source 13 is configured to emit illumination light to obtain either a bright pupil image or a dark pupil image. The calculation unit 23 sequentially detects the position of the right pupil based on the right pupil image data and sequentially detects the position of the left pupil based on the left pupil image data. The lighting control unit 21 adjusts the light intensity of the light source 13 at the timing of the third frame based on the brightness of the eye area corresponding to the right pupil detected from the right pupil image data of the first frame, and adjusts the light intensity of the light source 13 at the timing of the fourth frame based on the brightness of the eye area corresponding to the left pupil detected from the left pupil image data of the second frame. Even with this first modified example, the brightness of the pupils to be detected in the right pupil image data and the left pupil image data can be adjusted to the desired brightness, thereby further stabilizing the detection accuracy of the positions of the left and right pupils.

[0052] In the above-described embodiment, the bright pupil image and the dark pupil image are acquired as image data of separate frames. However, in the second modification example, the bright pupil image and the dark pupil image may be acquired as image data of the same frame. In the second modification example, the lighting control unit 21 controls the lighting timings of the light emitting elements 13a and 13b so that they simultaneously irradiate illumination light in synchronization with the timing of one frame. Further, the camera 10 is a so-called multi-spectral camera, has sensitivity to the wavelength band of the illumination light of the light emitting element 13a and the wavelength band of the illumination light of the light emitting element 13b, and outputs, as the bright pupil image and the dark pupil image, image data obtained by imaging images of light in each wavelength band at the same time. In this case as well, the lighting control unit 21 has a function of performing light amount adjustment for each of one camera 10 included in the pair of cameras 10 L and one camera 10 R . In this case, the shooting timing and the lighting timing in the camera 10 L and the shooting timing and the lighting timing in the camera 10 R are set to be shifted within the period of one frame.

[0053] Figure 9 is a conceptual diagram showing the temporal relationship of consecutive frames of image data acquired by the image acquisition unit 22 in the second modified example. In this way, the image acquisition unit 22 sequentially acquires the right pupil image data of the right pupil acquisition frame (first frame) for detecting the right pupil, the right pupil image data of the left pupil acquisition frame (second frame) for detecting the left pupil, the right pupil image data of the right pupil acquisition frame (third frame), and the right pupil image data of the left pupil acquisition frame (fourth frame). In this second modified example, the right pupil image data includes both the right bright pupil image and the right dark pupil image, and the left pupil image data includes both the left bright pupil image and the left dark pupil image. The calculation unit 23 then uses the first to fourth difference images calculated from the image data of the first to fourth frames to detect either the position of the right pupil or the position of the left pupil. The lighting control unit 21 adjusts the light intensity of the light-emitting elements 13a and 13b at the timing of the third frame based on the brightness of the eye area corresponding to the right pupil detected from the right pupil image data of the first frame, and adjusts the light intensity of the light-emitting elements 13a and 13b at the timing of the fourth frame based on the brightness of the eye area corresponding to the left pupil detected from the left pupil image data of the second frame. This second modification also makes it possible to adjust the brightness of the pupil to be detected in the difference image to the desired brightness, thereby further stabilizing the detection accuracy of the positions of the left and right pupils.

[0054] In the embodiment described above, the calculation unit 23 detected the position of the pupil based on a difference image calculated from the light pupil image and the dark pupil image. However, as a third modification, the position of the pupil may be detected using a calculated image obtained by calculating the image data of three frames. This third modification employs the method described in Japanese Patent Application Publication No. 2022-131345, in which the calculation unit 23 calculates the dark pupil image, light pupil image, and dark pupil image acquired in time series to obtain a multiplied image, and detects the position of the pupil based on the multiplied image.

[0055] Figures 10 and 11 are conceptual diagrams showing the temporal relationship of consecutive frames of image data acquired by the image acquisition unit 22 according to the third modified example. In this way, the image acquisition unit 22 acquires either the right bright pupil image or the right dark pupil image in the right pupil A acquisition frame, acquires the other right bright pupil image or the right dark pupil image in the subsequent right pupil B acquisition frame, and acquires either the right bright pupil image or the right dark pupil image again in the subsequent right pupil A acquisition frame. The calculation unit 23 detects the position of the right pupil using the multiplied image obtained by processing these three images. Also, the image acquisition unit 22 acquires either the left bright pupil image or the left dark pupil image in the left pupil C acquisition frame, acquires the other left bright pupil image or the left dark pupil image in the subsequent left pupil D acquisition frame, and acquires either the left bright pupil image or the left dark pupil image again in the subsequent left pupil C acquisition frame. The calculation unit 23 detects the position of the left pupil using the multiplied image obtained by processing these three images. As shown in Figure 10, the lighting control unit 21 adjusts the light intensity of the light-emitting element 13a or 13b in the immediately following right pupil B acquisition frame, right pupil A acquisition frame, and right pupil B acquisition frame based on the brightness of the eye area corresponding to the right pupil detected based on the multiplication image calculated from the three right pupil images, and adjusts the light intensity of the light-emitting element 13a or 13b in the immediately following left pupil D acquisition frame, left pupil C acquisition frame, and left pupil D acquisition frame based on the brightness of the eye area corresponding to the left pupil detected based on the multiplication image calculated from the three left pupil images. As shown in Figure 11, the lighting control unit 21 may also adjust the light intensity of the light-emitting element 13a or 13b in the immediately following right pupil B acquisition frame based on the brightness of the eye area corresponding to the right pupil detected based on the multiplication image calculated from the three right pupil images, or it may adjust the light intensity of the light-emitting element 13a or 13b in the immediately following left pupil D acquisition frame based on the brightness of the eye area corresponding to the left pupil detected based on the multiplication image calculated from the three left pupil images. This third modification also allows the brightness of the pupil to be detected in the multiplication image to be adjusted to the desired brightness, thereby further stabilizing the detection accuracy of the positions of the left and right pupils.

[0056] Furthermore, in the embodiments, second and third modifications described above, an unlit image may be acquired in addition to the bright pupil image and the dark pupil image by employing the method described in Japanese Patent No. 5145555. This unlit image only needs to be acquired in at least one frame corresponding to the four images: the right bright pupil image, the left bright pupil image, the right dark pupil image, and the left dark pupil image.

[0057] Furthermore, the embodiments, first modification, second modification, and third modification described above may also employ the method described in Japanese Patent No. 6083761 to estimate the image position of each pupil, the right pupil and the left pupil, in order to improve pupil detection accuracy. In the embodiments, first modification, second modification, and third modification described above, the position of one pupil is detected at a given frame. In the calculation unit 23 of the fourth modification, the position of the other pupil is tracked under the constraint that the distance between the left and right pupils is constant, and the pupil position detection accuracy can be improved by setting a window on the image based on the tracked position. However, if both pupils are detected in a given frame, the calculation unit 23 may track the position and angle of a vector connecting the positions of both pupils based on them, and set a window on the image based on the tracking result. In the fourth modification, the position of the left and right pupils may be tracked between frames using the distance between the rotation centers of both eyeballs, employing the method described in Japanese Patent No. 7030317. Alternatively, the position of the left and right pupils may be tracked between frames, assuming that the position and angle of the triangle formed by the internostril center and the left and right pupils change at a constant velocity. Alternatively, the position of the left and right pupils may be tracked between frames based on the detection position of one or more markers attached to the tip of the subject's nose instead of the internostril center.

[0058] Furthermore, the embodiments described above, including the first to fourth modified examples, are not limited to detecting the left and right pupils of subject A, but may be configured to detect the positions of three or more pupils of multiple subjects.

[0059] Furthermore, in the embodiments described above, and in the first to fourth modified examples, a light source capable of illuminating from within the aperture of a camera, as described in Japanese Patent Publication No. 4528980, may be used as the light source. [Explanation of symbols]

[0060] 1...Pupil detection device, 10...Camera, 12...Aperture, 13...Light source, 13a...Light-emitting element (first light source), 13b...Light-emitting element (second light source), 20...Image processing device, 21...Lighting control unit (light source control unit), 23...Calculation unit, A...Subject.

Claims

1. A camera that captures eye images in a series of frames by imaging the subject's eyes, A light source is provided in the camera to illuminate the pupil of the subject, The light source control unit for controlling the light intensity of the light source, The system includes a calculation unit that detects the positions of the left and right pupils of the subject based on the aforementioned eye image, The calculation unit detects at least one of the left and right pupils based on the eye image of the first frame, at least the other of the left and right pupils based on the eye image of the second frame following the first frame, at least one of the left pupils based on the eye image of the third frame following the first frame, and at least the other pupil based on the eye image of the fourth frame following the second frame. The light source control unit adjusts the amount of light at the timing of acquiring the eye image of the third frame based on the brightness of the eye portion corresponding to one pupil detected from the eye image of the first frame. Based on the brightness of the eye portion corresponding to the other pupil detected from the eye image of the second frame, the amount of light at the timing of acquiring the eye image of the fourth frame is adjusted. Pupil detection device.

2. The light source includes a first light source provided on the camera that makes the subject's pupil appear relatively bright to the camera, and a second light source provided on the camera that makes the subject's pupil appear relatively dark to the camera. The calculation unit detects at least one pupil based on a first calculated image obtained by calculating the eye image of the first bright pupil acquisition frame and the eye image of the first dark pupil acquisition frame, and detects at least the other pupil based on a second calculated image obtained by calculating the eye image of the second bright pupil acquisition frame after the first bright pupil acquisition frame and the eye image of the second dark pupil acquisition frame after the first dark pupil acquisition frame. Based on a third calculated image obtained by calculating the eye image of the third bright pupil acquisition frame after the first bright pupil acquisition frame and the eye image of the third dark pupil acquisition frame after the first dark pupil acquisition frame, at least one pupil is detected, and based on a fourth calculated image obtained by calculating the eye image of the fourth bright pupil acquisition frame after the third bright pupil acquisition frame and the eye image of the fourth dark pupil acquisition frame after the third dark pupil acquisition frame, at least the other pupil is detected. The light source control unit illuminates the first light source in the first bright pupil acquisition frame, the second bright pupil acquisition frame, the third bright pupil acquisition frame, and the fourth bright pupil acquisition frame, and illuminates the second light source in the first dark pupil acquisition frame, the second dark pupil acquisition frame, the third dark pupil acquisition frame, and the fourth dark pupil acquisition frame. In the third bright pupil acquisition frame or the third dark pupil acquisition frame, the light intensity of the first light source or the second light source is adjusted based on the brightness of the eye portion corresponding to one of the pupils detected from the first calculated image. In the fourth bright pupil acquisition frame or the fourth dark pupil acquisition frame, the light intensity of the first light source or the second light source is adjusted based on the brightness of the eye portion corresponding to the other pupil detected from the second calculated image. The pupil detection device according to claim 1.

3. The light source is provided in the camera and simultaneously emits a first light component that makes the subject's pupil appear relatively bright to the camera and a second light component that makes the subject's pupil appear relatively dark to the camera. The calculation unit obtains a first calculated image by calculating a bright pupil image in which one pupil is shown relatively brightly and a dark pupil image in which one pupil is shown relatively darkly, based on the eye image of the first frame, and detects at least one pupil based on the first calculated image. Based on the eye image of the second frame, a second calculated image is obtained by calculating a bright pupil image in which the other pupil is relatively brighter and a dark pupil image in which the other pupil is relatively darker, and the other pupil is detected based on the second calculated image. Based on the eye image of the third frame, a third calculated image is obtained by calculating the bright pupil image and the dark pupil image of the one pupil, and at least one pupil is detected based on the third calculated image. Based on the eye image of the fourth frame, a fourth calculated image is obtained by calculating the bright pupil image and the dark pupil image of the other pupil, and the other pupil is detected based on the fourth calculated image. The light source control unit adjusts the amount of light at the timing of acquiring the eye image of the third frame, based on the brightness of the eye portion corresponding to one pupil in the first calculated image acquired from the eye image of the first frame. Based on the brightness of the eye portion corresponding to the other pupil in the second calculated image obtained from the eye image of the second frame, the amount of light at the timing of obtaining the eye image of the fourth frame is adjusted. The pupil detection device according to claim 1.

4. The calculation unit acquires the first to fourth calculated images as difference images, respectively. The pupil detection device according to claim 2 or 3.

5. The light source control unit adjusts the amount of light based on the brightness of one pupil or the other pupil in the difference image. The pupil detection device according to claim 4.

6. The light source control unit adjusts the amount of light by changing the emission time width. The pupil detection device according to claim 1.

7. The light source is provided outside the aperture of the camera. The pupil detection device according to any one of claims 1 to 3.

8. A pupil detection method using a camera that acquires eye images in a series of frames by imaging the eyes of a subject, a light source provided on the camera that illuminates the pupil of the subject, a processing device that controls the camera and the light source, and performs image processing to detect the positions of the left and right pupils of the subject, Based on the eye image of the first frame, at least one of the left and right pupils is detected; based on the eye image of the second frame following the first frame, at least the other of the left and right pupils is detected; based on the eye image of the third frame following the first frame, at least one of the pupils is detected; and based on the eye image of the fourth frame following the second frame, at least the other pupil is detected. Based on the brightness of the eye portion corresponding to one pupil detected from the eye image of the first frame, the light intensity of the light source at the timing of acquiring the eye image of the third frame is adjusted. Based on the brightness of the eye portion corresponding to the other pupil detected from the eye image of the second frame, the amount of light at the timing of acquiring the eye image of the fourth frame is adjusted. Pupil detection method.

Images