Pupil detection device

Abstract

Provided is a pupil detection device capable of easily increasing the pupil detection accuracy irrespective of the state of a subject. A pupil detection device 1 is provided with: a light source 3A having a first central wavelength and provided at an opening 8 of a camera 2; a light source 3B having a second central wavelength and provided outside the light source 3A; and a control device 4. The camera 2 has: an optical element 9 provided with a division element 9A that allows light having the first central wavelength to pass therethrough and and allows transmission of linearly polarized light having a predetermined angle, and a division element 9B that allows light having the second central wavelength to pass therethrough and allows transmission of linearly polarized light having an angle different from the predetermined angle; and an imaging element 6 in which polarizers that allow transmission of linearly polarized light beams having four different angles are attached to respective pixels adjacent to each other. The control device 4 calculates a luminance corresponding to light having the first central wavelength on the basis of the luminances of the adjacent pixels to obtain a bright pupil image, calculates a luminance corresponding to light having the second central wavelength on the basis of the luminances of the adjacent pixels to obtain a dark pupil image, and calculates the position of a pupil image of a subject on the basis of a differential image obtained by comparing the bright pupil image with the dark pupil image.

Description

[Technical field]

[0001] The embodiment relates to a pupil detection device that detects pupils from an image of a person. [Background technology]

[0002] In recent years, devices that detect the position of a person's pupil from an image obtained using a light source such as a near-infrared light source and a video camera for the purpose of gaze detection, etc., are becoming popular (see Patent Document 1 below). In this device, a light that tends to make the pupil relatively bright is irradiated onto the face of a subject to obtain an image (bright pupil image), and a light that tends to make the pupil relatively dark is irradiated onto the face of the subject to obtain an image (dark pupil image). The subject's pupil is then detected by calculating a difference image using these images.

[0003] In such a method of pupil detection based on a difference image, there is generally a time difference between the timing of acquiring the bright pupil image and the dark pupil image, so if the subject's head moves between the acquisition of the two images, the pupil also moves, causing a positional shift between the pupil image in the bright pupil image and the pupil image in the dark pupil image, resulting in a limit to the accuracy of pupil detection. As a method for solving this problem, a method is used in which a corneal reflection image is detected from the bright pupil image and the dark pupil image, and a difference image is generated after position correction based on the position of the corneal reflection image (see Patent Document 2 below). In addition, a method of generating a difference image using the position of the nostrils is also used (see Patent Document 3 below). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-185431 A [Patent Document 2] JP 2008-29702 A [Patent Document 3] JP 2007-268026 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the conventional method described above, when the subject rotates his / her eyeball at high speed, the shape of the pupil itself shown in the image changes, and therefore even if the bright pupil image and the dark pupil image are aligned, the pupil images do not match between the two images, and therefore the accuracy of pupil detection when using the difference image is insufficient. It is possible to use a high-speed camera to minimize the time difference between the bright pupil image and the dark pupil image, but introducing a high-speed camera may be difficult in terms of cost.

[0006] The present embodiment has been made in consideration of the above-mentioned problems, and aims to provide a pupil detection device that can easily improve the accuracy of pupil detection regardless of the condition of a subject. [Means for solving the problem]

[0007] In order to solve the above problems, a pupil detection device according to one embodiment of the present disclosure includes a camera that captures an image of a subject's eye by capturing an image of the subject's eye, a first light source that is provided on the outside or inside of an opening of the camera and illuminates the pupil of the subject relative to the camera with light of a first central wavelength, a second light source that is provided on the outside or inside of the opening of the camera and illuminates the pupil of the subject relative to the camera with a second central wavelength different from the first central wavelength, and a calculation device that processes the eye image, and the camera includes an image sensor in which a polarizer that transmits linearly polarized light of at least two different angles is attached to each of adjacent pixels, and a first dividing element that combines a bandpass filter that transmits light of the first central wavelength and a first polarizer that transmits linearly polarized light of a predetermined angle. and a second dividing element combining a bandpass filter that passes light of a second central wavelength and a second polarizer that transmits linearly polarized light at an angle different from the predetermined angle, and an optical element divided and provided along the opening between the opening and the image sensor, wherein the arithmetic device calculates a luminance corresponding to the light of the first central wavelength based on the luminance of adjacent pixels in the eye image, combines the calculated luminances to obtain a bright pupil image in which the subject's pupil is relatively bright, calculates a luminance corresponding to the light of the second central wavelength based on the luminance of adjacent pixels in the eye image, combines the calculated luminances to obtain a dark pupil image in which the subject's pupil is relatively dark, and calculates the position of the subject's pupil image based on a comparison image obtained by comparing the bright pupil image with the dark pupil image.

[0008] The "camera opening" here means a portion for taking in the light of an image from outside the camera to the image sensor inside the camera, and is not necessarily limited to the circular lens part of the camera barrel, but means the opening when the lens part of the camera is covered with a cover member having an opening narrower than the lens part. The shape of the opening is not limited to a circle, and may be various shapes such as a polygon such as a rectangle, an ellipse, etc.

[0009] According to the pupil detection device of the above embodiment, the light from the pupil of the subject illuminated with the light of the first central wavelength by the first light source is converted into linearly polarized light at a predetermined angle by passing through the first division element of the optical element, and the luminance is detected by adjacent pixels of the image sensor to which two types of polarizers are attached. At the same time, the light from the pupil of the subject illuminated with the light of the second central wavelength by the second light source is converted into linearly polarized light at an angle different from the predetermined angle by passing through the second division element of the optical element, and the luminance is detected by adjacent pixels of the image sensor to which two types of polarizers are attached. Here, the image of the pupil illuminated with the light of the first central wavelength is relatively brighter than the image of the pupil illuminated with the light of the second central wavelength. The computing device then calculates a luminance corresponding to the light of the first central wavelength from the luminance of adjacent pixels detected by the image sensor and combines the luminances to obtain a bright pupil image, calculates a luminance corresponding to the light of the second central wavelength from the luminance of adjacent pixels detected by the image sensor and combines the luminances to obtain a dark pupil image, and calculates the position of the pupil image based on the bright pupil image and the dark pupil image. This makes it possible to easily improve the detection accuracy of the pupil image regardless of the state of the subject (for example, even if the subject's eyeball rotates), because there is no difference in the timing at which the bright pupil image and the dark pupil image are obtained. Effect of the Invention

[0010] According to the embodiment, it is possible to easily improve the pupil detection accuracy regardless of the condition of the subject. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a schematic configuration of a pupil detection device according to a first embodiment. [Diagram 2] 2 is a plan view of the lighting device of FIG. 1 as viewed from outside the housing. [Diagram 3] 2 is a plan view of the optical element in FIG. 1 as viewed from the outside of the housing. [Figure 4] 2 is a plan view of the optical element in FIG. 1 as viewed from the outside of the housing. [Diagram 5]FIG. 2 is a plan view of the imaging element of FIG. [Figure 6] 2 is a block diagram showing a hardware configuration of a control device in FIG. 1. [Figure 7] 2 is a block diagram showing a functional configuration of the pupil detection device in FIG. 1. [Figure 8] 1 is a graph showing the relationship between pupil area A and pupil luminance LB in a bright pupil image and pupil luminance LD in a dark pupil image. [Figure 9] 13 is a graph showing the relationship between pupil area A and pupil luminance LS in a difference image. [Figure 10] 1 is a graph showing the relationship between the distance from the center of the camera aperture to the light source and pupil luminance. [Figure 11] 13 is a plan view showing the configuration of a light source and an optical element according to a modified example. FIG. [Figure 12] 13 is a plan view showing the configuration of a light source and an optical element according to a modified example. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the pupil detection device according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are given the same reference numerals, and duplicated explanations will be omitted.

[0013] (First embodiment)

[0014] As shown in FIG. 1, the pupil detection device 1 according to the first embodiment of the present invention includes a camera 2, an illumination device (light source) 3, and a control device (arithmetic device) 4. The camera 2 includes a housing 5, an imaging element (image sensor) 6 such as a CCD or CMOS housed in the housing 5, and an objective lens 7 and an optical element 9 housed in the housing 5. The camera 2 may be a high-speed camera in which the time interval for acquiring one frame of an image is very short, or may be a so-called medium-speed camera, a low-speed camera, or a camera having a frame rate of about 60 Hz or 30 Hz. The housing 5 has a circular opening 8 formed on a surface facing the eyeball A of the subject. The objective lens 7 and the optical element 9 are disposed between the opening 8 inside the housing 5 and the imaging element 6. The optical axis L0 of the objective lens 7 coincides with the central axis of the opening 8. The imaging element 6 is fixed so that its light receiving surface perpendicularly intersects with the optical axis L0 of the objective lens 7. The optical element 9 is disposed inside the objective lens 7 (on the imaging element 6 side) on the optical axis L0 of the objective lens 7. The imaging element 6 captures an image of the eyeball A of the subject to be observed, thereby generating eye image data and outputting the data to the control device 4. The control device 4 controls the light emission intensity, lighting timing, and lighting period (light emission period) of the illumination device 3, as well as the imaging timing and imaging period of the camera 2. The control device 4 also executes image generation processing, comparison processing, pupil detection processing, and corneal reflex detection processing based on the eye image data output from the imaging element 6. That is, the control device 4 also functions as a pupil detection means and a corneal reflex detection means.

[0015] The diameter of the opening 8 is smaller than the diameter of the objective lens 7 and is approximately the same as the effective diameter of the objective lens 7. Moreover, the optical element 9 has an approximately circular shape overall, and has a diameter approximately the same as the effective diameter of the objective lens 7. With this configuration, an image of the vicinity of the eyeball A of the subject is introduced toward the objective lens 7 and the image sensor 6 in the camera 2 through the opening 8, and then is imaged so as to converge on the light receiving surface of the image sensor 6 by an optical system including the objective lens 7 and the optical element 9 in the camera 2.

[0016] The illumination device 3 emits illumination light toward the face of the subject. As shown in Fig. 2, the illumination device 3 has a casing 10 and light sources 3A and 3B embedded in the casing 10. The casing 10 is attached to the housing 5 so as to cover the outside of the opening 8 along the edge of the opening 8. The light sources 3A and 3B are both provided on the casing 10 so as to emit illumination light along the optical axis L0 of the objective lens 7, and are configured to be point symmetric with respect to the center of the opening 8.

[0017] The light source (first light source) 3A is a light source for illuminating the eye (pupil) of the subject with illumination light (first illumination light) for obtaining a bright pupil image. The bright pupil image refers to an image in which the pupil of the subject is relatively brighter than a dark pupil image described later. The light source 3A is, for example, made up of a plurality of semiconductor light-emitting elements (LEDs) whose output light has a central wavelength (first central wavelength) in the near-infrared region, and is disposed at a first distance D1 that is relatively close from the center of the opening 8. Specifically, the light-emitting elements constituting the light source 3A are disposed on the casing 10 in a ring shape at equal intervals along the edge of the opening 8 outside the opening 8. It is preferable that the light source 3A is disposed as close as possible to the edge of the opening 8. As a result, as described later, in the image of the subject illuminated by the light source 3A, the pupil is projected brighter, and even a small pupil can be easily detected.

[0018] The light source (second light source) 3B is a light source for illuminating the eye (pupil) of the subject with illumination light (second illumination light) for obtaining a dark pupil image. The dark pupil image refers to an image in which the pupil of the subject is relatively dark compared to the above-mentioned bright pupil image. The light source 3B is, for example, made up of a plurality of semiconductor light-emitting elements (LEDs) in the near-infrared region whose output light has a central wavelength (second central wavelength) longer than the first central wavelength, and is disposed at a second distance D2, which is relatively far from the center of the opening 8. This second distance D2 is greater than the first distance D1. Specifically, the light-emitting elements constituting the light source 3B are disposed continuously in a ring shape at equal intervals on the casing 10, spaced from the light source 3A to the outside of the opening 8.

[0019] When illumination light is emitted from the light source 3A to the eyeball A of the subject, and the pupil illuminated by the illumination light is captured by the camera 2, a bright pupil image is obtained. When illumination light is emitted from the light source 3B to the eyeball A of the subject, and the pupil illuminated by the illumination light is captured by the camera 2, a dark pupil image is obtained. This is due to the following property. In other words, when illumination light to eyeball A is incident from a position relatively distant from the optical axis L0 of camera 2, the illumination light that enters through the pupil of eyeball A, is reflected inside the eyeball, and passes through the pupil again is unlikely to reach camera 2, so the pupil appears relatively dark.

[0020] Here, the first central wavelength of the output light of the light source 3A is set to, for example, 850 nm, and the second central wavelength of the output light of the light source 3B is set to, for example, 940 nm, which is longer than the first central wavelength. However, the light source 3A, which is a light source for acquiring a bright pupil image, may be a light source having a wavelength of output light shorter than about 900 nm in that the luminance of the light reflected back from the retina is strong. Similarly, the light source 3B, which is a light source for acquiring a dark pupil, may be a light source having a wavelength of output light longer than about 900 nm in that the luminance of the light reflected back from the retina is weak. On the other hand, since the light emission power of a light source with a long wavelength is weak and the sensitivity of a camera generally decreases as the wavelength becomes longer, it is preferable that the light source 3B has about twice as many light-emitting elements as the light source 3A. When there is no room to arrange a large number of light-emitting elements, the light source 3B may have a structure in which light-emitting elements are arranged in a double ring shape.

[0021] The light emission intensities (light emission powers) of the light sources 3A and 3B are preset so that the illuminance on the face of the subject of observation, who is the subject of photography, is approximately the same when the light sources 3A and 3B are turned on for the same lighting period. For this purpose, for example, the current value or power value supplied from the control device 4 to the light sources 3A and 3B is preset.

[0022] FIG. 3 shows the structure of the optical element 9 arranged inside the opening 8. The optical element 9 has a disk-like structure and is arranged so that the optical axis L0 of the objective lens 7 passes through the vicinity of its center and is approximately perpendicular to the optical axis L0. This optical element 9 is composed of semicircular plate-like splitting elements 9A and 9B split into two at the center. The splitting element 9A is an optical element in which a bandpass filter that passes a first illumination light of a first center wavelength and a first polarizer that transmits linearly polarized light with a polarization direction of 0 degrees based on the vertical direction of the opening 8 are combined in two layers. The splitting element 9B is an optical element in which a bandpass filter that passes a second illumination light of a second center wavelength and a second polarizer that transmits linearly polarized light with a polarization direction of 90 degrees based on the vertical direction are combined in two layers. That is, the polarization direction of the linearly polarized light transmitted by the first polarizer and the polarization direction of the linearly polarized light transmitted by the second polarizer are approximately perpendicular to each other. The optical element 9 having such a structure is configured so that the splitting elements 9A and 9B are arranged on both sides of a central axis (line) that passes horizontally through the center of the opening surface of the opening 8, more specifically, so as to be linearly symmetrical with respect to the central axis. The light sources 3A and 3B of the illumination device 3 are configured in a similar manner. Note that the shape of the optical element 9 combining the splitting elements 9A and 9B is not limited to a disk shape, and may be any other shape, such as a rectangle, as long as it corresponds to the shape of the edge of the opening 8.

[0023] On the other hand, the optical element 9 may have a structure as shown in FIG. 4. The optical element 9 shown in FIG. 4 is composed of splitting elements 9A and 9B having a continuous shape corresponding to the shape of the edge of the opening 8, and these splitting elements 9A and 9B have a shape divided into two by a boundary line along the edge of the opening 8, and the splitting element 9B is disposed inside the splitting element 9A. That is, the splitting element 9B has a disk-like shape (circular shape) with its center located near the optical axis L0, and the splitting element 9A has a ring-like shape located outside the splitting element 9B. The optical element 9 having such a structure is configured such that the splitting elements 9A and 9B are point-symmetric with respect to the center of the opening 8, similar to the illumination device 3. The light sources 3A and 3B of the illumination device 3 are also configured in the same manner. Note that the shape of the optical element 9 formed by combining the splitting elements 9A and 9B is not limited to a disk-like shape, and may have other shapes such as a rectangular shape as long as it corresponds to the shape of the edge of the opening 8.

[0024] FIG. 5 shows the structure of the image sensor 6 arranged inside the opening 8, as viewed from the opening 8 side. The image sensor 6 has a light receiving surface 20 arranged to be approximately perpendicular to the optical axis L0, a plurality of pixels 21 with light receiving elements (not shown) formed therein and arranged in a two-dimensional array on the light receiving surface 20, and four types of polarizers 22A, 22B, 22C, and 22D arranged on the surfaces of the plurality of pixels 21 so as to cover the pixels 21. That is, different types of polarizers 22A, 22B, 22C, and 22D are attached to the surfaces of four adjacent pixels 21 (hereinafter also referred to as a four-pixel group). The polarizer 22A is an optical element that transmits linearly polarized light with a polarization direction of 0 degrees relative to the vertical direction of the opening 8. The polarizer 22B is an optical element that transmits linearly polarized light with a polarization direction rotated 45 degrees counterclockwise relative to the vertical direction of the opening 8. Polarizer 22C is an optical element that transmits linearly polarized light with a polarization direction rotated 90 degrees counterclockwise with respect to the up-down direction of opening 8. Polarizer 22D is an optical element that transmits linearly polarized light with a polarization direction rotated 135 degrees counterclockwise with respect to the up-down direction of opening 8. The imaging element 6 has a structure in which such structures are repeated two-dimensionally (in the left-right and up-down directions in FIG. 5).

[0025] The range of the four pixel groups on the light receiving surface 20 of the image sensor 6 is sufficiently small compared with the size of the objective lens 7. Therefore, the optical element 9 and the image sensor 6 are configured so that the first illumination light passing through the dividing element 9A of the optical element 9 is incident on the four pixels constituting the four pixel groups with the same intensity, and the second illumination light passing through the dividing element 9B of the optical element 9 is incident on the four pixels constituting the four pixel groups with the same intensity.

[0026] Next, the configuration of control device 4 included in pupil detection device 1 will be described with reference to FIGS.

[0027] The control device 4 may be a computer that controls the image sensor 6 and the light sources 3A, 3B, and processes the eye image data of the subject to be observed to detect the pupil and corneal reflex. The control device 4 may be constructed by a stationary or portable personal computer (PC), a workstation, or other types of computers. Alternatively, the control device 4 may be constructed by combining multiple computers of any type. When multiple computers are used, these computers are connected via a communication network such as the Internet or an intranet.

[0028] As shown in FIG. 6, the control device 4 includes a CPU (processor) 101, a main memory unit 102, an auxiliary memory unit 103, a communication control unit 104, an input device 105, and an output device 106. The CPU 101 executes an operating system, application programs, etc. The main memory unit 102 is composed of a ROM and a RAM. The auxiliary memory unit 103 is composed of a hard disk, a flash memory, etc. The communication control unit 104 is composed of a network card or a wireless communication module. The input device 105 includes a keyboard, a mouse, etc. The output device 106 includes a display, a printer, etc.

[0029] Each functional element of the control device 4, which will be described later, is realized by loading predetermined software onto the CPU 101 or main memory unit 102, operating the communication control unit 104, input device 105, output device 106, etc. under the control of the CPU 101, and reading and writing data in the main memory unit 102 or auxiliary memory unit 103. Data and databases necessary for processing are stored in the main memory unit 102 or auxiliary memory unit 103.

[0030] As shown in FIG. 7, the control device 4 has an image sensor driving unit 11, a lighting control unit 12, and a detection unit 13 as functional components. The image sensor driving unit 11 is a functional element that controls the timing of capturing images of the image sensor 6. Specifically, the image sensor 6 is controlled to capture images repeatedly at a predetermined frame rate and a predetermined exposure time, and to continuously acquire eye image data representing an image of the eyeball A of the observation subject. The lighting control unit 12 is a functional element that controls the lighting timing so that the light sources 3A and 3B are simultaneously turned on in accordance with the capturing of the image sensor 6, and also controls the amount of light emitted from the light sources 3A and 3B. At this time, in order to improve the accuracy of the detection position of the pupil or corneal reflex when the head of the observation subject moves, it is preferable that the lighting control unit 12 turns on the light sources 3A and 3B for the same period. The detection unit 13 is a functional element that uses the eye image data output from the image sensor 6 to detect the pupil and corneal reflex in the eye image data. There is no limitation on the output destination of the information on the detected pupil and corneal reflex. For example, the control device 4 may display the results as images, graphics, or text on a monitor, store them in a storage device such as a memory or database, or transmit them to another computer system via a communications network.

[0031] The detection unit 13 has, as functional components, an image acquisition section 14, an image calculation section 16, a difference image generation section 17, a pupil detection section 18, and a corneal reflection detection section 19. The image acquisition section 14 acquires eye image data that is continuously photographed (acquired) at a predetermined frame rate from the image sensor 6. The image calculation section 16 calculates the luminance value (light intensity) of each pixel of the bright pupil image based on the eye image data at each imaging timing, and acquires the bright pupil image by combining these luminance values. The image calculation section 16 also calculates the luminance value (light intensity) of each pixel of the dark pupil image based on the eye image data at each imaging timing, and acquires the dark pupil image by combining these luminance values. The difference image generation section 17 generates a difference image as a kind of comparison image obtained by comparing the bright pupil image and the dark pupil image. Specifically, the difference image generation section 17 generates a difference image obtained by comparing both images by calculating the difference in luminance between corresponding pixels of the bright pupil image and the dark pupil image. The pupil detection unit 18 is a functional element that calculates the position of the pupil image using the difference image. The corneal reflection detection unit 19 is a functional element that calculates the position of the corneal reflection image using the bright pupil image or the dark pupil image. An example of the processing performed by the pupil detection unit 18 and the corneal reflection detection unit 19 is as follows. First, the pupil detection unit 18 binarizes the difference image based on a pupil threshold, removes isolated points, removes noise by morphological processing, and performs labeling. Then, the pupil detection unit 18 detects the pixel group having the shape most similar to a pupil as the pupil. At this time, even if the pupil is hidden by the eyelids or eyelashes, the boundary between the eyelids or eyelashes and the pupil is excluded as a false pupil contour, and only the true pupil contour is fitted to an ellipse to detect the position of the true pupil contour on the difference image, and the center position of the pupil image is calculated from the equation of the ellipse obtained by ellipse fitting. The corneal reflection detection unit 19 also binarizes the bright pupil image near the pupil with a corneal reflection threshold value higher than the pupil luminance, and determines the center of the corneal reflection image as the center of gravity taking the luminance into consideration. The pupil luminance is given by the average luminance of the pixels constituting the binarized pupil, not the area of ​​the ellipse obtained as a result of ellipse fitting. The corneal reflection detection unit 19 may calculate the position of the corneal reflection image using the dark pupil image as the target.

[0032] Next, the functional configuration of the image calculation unit 16 will be described in detail.

[0033] The principle of acquisition of bright and dark pupil images by the image calculation unit 16 is as follows. When acquiring eye image data, the light transmitted through the first and second gratings 9A and 9B is incident with the same light intensity on four pixel groups each consisting of four adjacent pixels 21. The light intensity of each incident light is expressed as I(λ a1 ),I(λ a2 ), the light intensities O detected by the pixels 21 to which the polarizers 22A, 22B, 22C, and 22D are attached among the four pixel groups are expressed as follows: i (i=1 to 4) is expressed by the following formula.

number

[0034] The above formula can be expressed as a determinant to obtain the following formula:

number

number

number

[0035] Using the above formula, the image calculation unit 16 calculates the luminance value [O i ] and the brightness value [O i ] and the preset pseudo-inverse matrix [M ij ] + Using the above, the luminance value I(λ) corresponding to the light of the first central wavelength incident on the four pixel groups is calculated. a1 ) and the luminance value I(λ a2 Then, the image calculation unit 16 calculates the luminance value I(λ a1 ) to generate two-dimensional image data, a bright pupil image is obtained, and the brightness value I(λ a2 ) to generate two-dimensional image data to obtain a dark pupil image.

[0036] The effects of pupil detection device 1 of the first embodiment will be described.

[0037] In the pupil detection device 1 of the embodiment described above, light from the pupil of the subject illuminated by the light source 3A with light of the first central wavelength is transmitted through the first divisional element 9A of the optical element 9 and converted into linearly polarized light at a predetermined angle, and its luminance value is detected by four adjacent pixels to which the four types of polarizers 22A, 22B, 22C, and 22D of the image sensor 6 are attached. At the same time, light from the pupil of the subject illuminated by the light source 3B with light of the second central wavelength is transmitted through the second divisional element 9B of the optical element 9 and converted into linearly polarized light at an angle different from the predetermined angle, and its luminance value is detected by four adjacent pixels to which the four types of polarizers 22A, 22B, 22C, and 22D of the image sensor 6 are attached. Here, the image of the pupil illuminated with light of the first central wavelength is relatively brighter than the image of the pupil illuminated with light of the second central wavelength. Then, control device 4 calculates a luminance value corresponding to the light of the first central wavelength from the luminance values ​​of four adjacent pixels detected by image sensor 6 and combines these luminance values ​​to obtain a bright pupil image, calculates a luminance value corresponding to the light of the second central wavelength from the luminance values ​​of adjacent pixels detected by image sensor 6 and combines these luminance values ​​to obtain a dark pupil image, and calculates the position of the pupil image based on the bright pupil image and the dark pupil image. This makes it possible to easily improve the detection accuracy of the pupil image regardless of the state of the observed subject (for example, even if the observed subject's eyeball rotates).

[0038] In particular, in this embodiment, the polarization direction of the light passing through the first dividing element 9A and the polarization direction of the light passing through the second dividing element 9B are approximately perpendicular to each other. In this case, the polarization directions of the light of the first central wavelength and the light of the second central wavelength incident on each pixel of the image sensor 6 are clearly separated, improving the accuracy of calculating the luminance of the bright pupil image and the dark pupil image, and as a result, improving the accuracy of detecting the position of the pupil image.

[0039] In addition, the image sensor 6 has polarizers 22A, 22B, 22C, and 22D that transmit linearly polarized light of four different angles attached to every four adjacent pixels, and the control device 4 calculates the luminance values ​​corresponding to the light of the first and second central wavelengths based on the luminance values ​​of the four adjacent pixels in the eye image data. In this case, it is possible to equalize the arrangement balance of the multiple pixels that detect the various polarization direction components of the light of the first and second central wavelengths on the light receiving surface 20. This makes it possible to equalize the bright pupil image and the dark pupil image, and stabilize the accuracy of the position of the detected pupil image.

[0040] In addition, the distance of light source 3A from the center of aperture 8 of camera 2 is made smaller than the distance of light source 3B from the center of aperture 8. With this structure, it is possible to increase the luminance difference between the image of the pupil illuminated with light of the first central wavelength and the image of the pupil illuminated with light of the second central wavelength. As a result, it is possible to further improve the detection accuracy of the pupil image position using the comparison image.

[0041] Furthermore, the optical element 9 is configured so that the first dividing element 9A and the second dividing element 9B are point-symmetric or line-symmetric about the opening 8, and the light source 3A and the light source 3B are configured so that they are point-symmetric or line-symmetric about the opening 8. This makes it possible to prevent uneven brightness of the pupil image and its surroundings in the bright pupil image and the dark pupil image, makes it possible to make only the pupil image stand out in the comparison image, and improves the accuracy of detecting the position of the pupil image using the comparison image.

[0042] In particular, when optical element 9 has a structure as shown in FIG. 3, it is possible to prevent unevenness in the brightness of the pupil image and its surrounding areas in the bright pupil image and dark pupil image in the vertical direction, thereby improving the accuracy of detecting the pupil image position using the comparison image. Furthermore, when optical element 9 has a structure as shown in FIG. 4, it is possible to prevent two-dimensional unevenness in the luminance of the pupil image and its surrounding areas in the bright pupil image and the dark pupil image, thereby further improving the detection accuracy of the pupil image position using the comparison image.

[0043] Furthermore, the control device 4 calculates the position of the corneal reflection of the subject caused by the illumination of the light source 3A or the light source 3B based on the bright pupil image or the dark pupil image. With this configuration, the position of the corneal reflection can be stably detected.

[0044] The advantages of the configurations of the illumination device 3 and the optical element 9 in this embodiment will be described.

[0045] In pupil detection based on a comparison image such as a difference image, the greater the difference in luminance in the pupil image between the bright pupil image and the dark pupil image, the higher the accuracy of pupil detection. Furthermore, if the pupil position does not change relative to the optical system (camera 2 and illumination device 3), the line of sight direction relative to the optical system does not change, and further, the amount of light in the illumination device 3 and the sensitivity of the camera 2 do not change (referred to as "pupil detection conditions"), the larger the pupil, the higher the luminance of the pupil in both the bright pupil image and the dark pupil image. Figure 8 shows the relationship between pupil area A and pupil luminance L in the bright pupil image. B and pupil luminance in the dark pupil image L D 9 is a graph showing the relationship between pupil area A and pupil luminance L in the difference image. S In this way, ideally, the following relationship holds: L B =k B A, L D =k D A, L S =k S A=(k B -k D )·A In the above formula, k B , k D , k Sare proportional constants, and are constants when the above-mentioned "pupil detection condition" is satisfied. As can be seen from this relational expression, when the pupil is large, the pupil luminance in the bright pupil image is high, and the pupil luminance is also high in the difference image. Therefore, since the luminance of the area other than the pupil is close to zero, it is easy to detect the pupil from the difference image. On the other hand, when the pupil is small, the light incident on the pupil is also reduced according to the pupil area, so the luminance of the pupil is low. As a result, when the pupil is small (for example, about 2.5 mm in diameter), it is necessary to irradiate the face with stronger light. In order to increase the amount of light from the light source, it is necessary to increase the current supplied to the light source or increase the number of light sources, which increases power consumption or costs. In addition, in order to increase the pupil luminance in the bright pupil image, it is necessary to place the light source as close as possible to the aperture of the camera, but in that case, there is a possibility that a problem will occur such as a lack of physical space to place the light source.

[0046] FIG. 10 is a graph showing the relationship between the distance from the center of the camera aperture to the light source (the difference in angle between the eye of the subject and the light source and the center of the aperture) and pupil luminance. B is the relationship when the pupil is large, graph G S shows the relationship when the pupil is small. Thus, the pupil luminance rises rapidly as the distance between the opening and the light source decreases from a large value. As described above, in order to increase the difference between the pupil luminance of the bright pupil image and the pupil luminance of the dark pupil image, it is preferable to position light source 3A as close as possible to the opening and light source 3B away from the opening. In this case, when the pupil is large, the difference in luminance between the bright pupil image and the dark pupil image is likely to be large, so problems are unlikely to occur. Problems are likely to occur when the pupil becomes extremely small. Even in such a case, it is desirable to increase the pupil luminance of the bright pupil image in order to more reliably detect the pupil from the difference image. This is because, when the pupil is small, the pupil luminance of the bright pupil image is small, as shown in graph G. S As shown in Fig. 1, if the light source 3B is moved slightly away from the aperture, the pupil luminance becomes sufficiently low (distance D A ), even if it is moved farther than necessary, the pupil luminance is hardly decreased. On the other hand, if the light source 3A is brought closer to the opening (for example, at a distance DB Distance D from C 2), the pupil luminance in the bright pupil image increases rapidly, and the accuracy of pupil detection from the difference image increases.

[0047] On the other hand, when the pupil is large, graph G B As shown in graph G, the pupil tends to be bright until light source 3B is quite far from the aperture. Since the pupil luminance in the bright pupil image is much higher than that in the dark pupil image, there is no problem in pupil detection unless the luminance value reaches a saturation level, since there is a luminance difference between the bright and dark pupil images. If it is desired to reduce the amount of light from the light sources for some reason, for example to reduce power consumption, it is possible to reduce the amount of light from light sources 3A and 3B in order to keep the pupil luminance constant when the pupil is large and when it is small. This results in a larger pupil, as shown in graph G. B and graph G S In this case, the luminance level for a large pupil approaches the luminance level for a small pupil.

[0048] Based on the above findings, this embodiment employs the configurations of the illumination device 3 and optical element 9 shown in Figures 2 and 3. By employing such a configuration, it is possible to dispose the light source 3A as close as possible to the opening 8, and it is possible to guide the light components of the first central wavelength and the second central wavelength of the pupil image from the subject to the imaging element 6 in the opening 8 in a well-balanced manner.

[0049] In addition, in this embodiment, when the configuration of the illumination device 3 and the optical element 9 shown in Fig. 2 and Fig. 4 is adopted, the light components of the first central wavelength and the light components of the second central wavelength of the pupil image from the observation subject can be guided to the image sensor 6 in the opening 8 in a well-balanced manner in the two-dimensional direction. In addition, this configuration allows the light source 3A to be arranged close to the opening 8. Since the splitting element 9A has a ring-like shape, when any one point of the splitting element 9A is used as a reference, some of the light sources 3A are far away. However, if there is even one light source 3A close to that point, the effect of brightening the pupil is very high, so that the light sources 3A as a whole have a sufficient effect of brightening the pupil. In addition, the light source 3B can be sufficiently separated from the splitting element 9B in the opening 8, and the pupil luminance of the dark pupil image can be sufficiently reduced.

[0050] The present invention is not limited to the above-described embodiment, and the configuration of the above-described embodiment can be modified in various ways.

[0051] In the above-described embodiment, the difference image generating unit 17 generates a difference image as a comparison image by comparing the bright pupil image and the dark pupil image, but a division image may be used as the comparison image as described in JP 2008-246004 A. Furthermore, an image may be generated as the comparison image by comparing the bright pupil image and the dark pupil image by other calculations.

[0052] In the above-described embodiment, the control device 4 may detect the gaze direction and gaze point of the observed subject based on the position of the pupil image detected by the pupil detection unit 18 based on the difference image and the position of the corneal reflection image detected by the corneal reflection detection unit 19 for each of the dark pupil image and the bright pupil image. The method for calculating the gaze direction and gaze point may employ a method developed by the present inventors (see International Publication WO2012 / 020760). By using the position of the pupil image detected by the control device 4 of this embodiment, the gaze direction and gaze point of the observed subject can be stably detected.

[0053] Furthermore, the configuration of each light source in the illumination device 3 according to the above-described embodiment and the configuration of the optical element 9 may be modified in various ways.

[0054] FIG. 11 shows a modified example of the arrangement of the light sources 3A, 3B and the splitting elements 9A, 9B. In this example, the light sources 3A, 3B and the splitting elements 9A, 9B are arranged on both sides of a central axis (line) that passes through the center of the opening 8 from top to bottom along the opening surface of the opening 8, more specifically, arranged so as to be line-symmetrical with respect to the central axis. The splitting element 9A has two divided shapes, which are arched shapes on the left and right, corresponding to the shape of the edge of the opening 8, provided on both sides of the central axis that passes through the center of the opening 8 from top to bottom. The splitting element 9A may have two or more divided shapes (for example, four). The splitting element 9B is arranged at the center of the opening 8, sandwiched between the arched shapes. The light source 3A is arranged outside the splitting element 9A near the opening 8, separated left and right, and the light source 3B is arranged outside the light source 3A, separated left and right. This configuration also has a left-right symmetrical configuration, so the luminance difference in the face region in the bright pupil image and the dark pupil image is kept small, and this is an effective configuration when it is desired to reduce the vertical size of the illumination device 3.

[0055] FIG. 12 shows another modified example of the arrangement of the light sources 3A, 3B and the splitting elements 9A, 9B. The splitting elements 9A, 9B have the same configuration as that of FIG. 3, and the light source 3A is arranged only near the opening 8 on the splitting element 9A side (left side), and the light source 3B is also arranged separately on the left side of the opening 8. In this configuration, the distance between the opening 8 on the splitting element 9A side and the light source 3A is short, and there is no light source 3A far from the opening 8 on the splitting element 9A side, so that the pupil luminance in the bright pupil image is kept high. Furthermore, the distance between the light source 3B and the light source 3A and the center of the opening 8 can be reduced while maintaining a certain distance from the opening 8 on the splitting element 9B side. As a result, the size of the entire optical system can be reduced. Even if a luminance gradient occurs in the face or inside the pupil in the bright pupil image and the dark pupil image, the positions of the light sources 3A and 3B are close to each other, so that the tendency of the luminance gradient does not differ greatly between the two images. Therefore, parts other than the pupil are less noticeable in the difference image, and the detection accuracy of the pupil can be maintained.

[0056] 11 are not limited to point-symmetric or line-symmetric configurations, and may be arranged and shaped differently from point-symmetric or line-symmetric with respect to the center or central axis of the aperture 8. For example, in order to adjust the luminance of the bright pupil image and the dark pupil image, the arrangement and shape may be adjusted from point-symmetric or line-symmetric.

[0057] The optical element 9 may be configured with a plurality of divided sections divided into a plurality of parts of any shape. For example, the optical element 9 may be configured with a plurality of rectangular divided sections. In such a configuration, the dividing elements 9A and 9B are appropriately assigned (for example, alternately assigned) to the plurality of divided sections.

[0058] Furthermore, the optical element 9 does not need to be disposed inside the objective lens 7 (on the imaging element 6 side), and may be disposed outside the objective lens 7 (on the side of the subject of observation). Alternatively, when the camera lens provided in the camera 2 is configured to include a plurality of optical lenses, the optical element 9 may be disposed between any two of the plurality of optical lenses, or may be disposed in front of or behind the entire camera lens (on the side of the subject of observation or on the side of the imaging element 6).

[0059] Furthermore, as described in Japanese Patent No. 4528980, the light sources 3A and 3B may be disposed inside the opening 8 of the camera 2. Furthermore, the light source 3A may be disposed inside the opening 8, and the light source 3B may be disposed outside the opening 8.

[0060] In the above embodiment, the image sensor 6 has a structure in which four types of polarizers are provided for every four pixel groups, but it is sufficient that the image sensor 6 has a structure in which at least two types of polarizers are provided. For example, two types of polarizers may be provided for every two adjacent pixels 21, or three types of polarizers may be provided for every three adjacent pixels 21. Even in such a modified example, the control device 4 can calculate the luminance value of the light component of the first central wavelength and the luminance value of the light component of the second central wavelength based on the pixel values ​​of two adjacent pixels or three adjacent pixels.

[0061] In addition, in the above embodiment, the control device 4 calculates the luminance value of the light component with the first central wavelength and the luminance value of the light component with the second central wavelength based on the pixel values ​​of four adjacent pixels, but it is also possible to calculate the luminance value of the light component with the first central wavelength and the luminance value of the light component with the second central wavelength based on the pixel values ​​of at least two pixels (two pixels, or three pixels) among the four adjacent pixels.

[0062] In the above embodiment, the polarization direction of the light passing through the first polarizer and the polarization direction of the light passing through the second polarizer may be substantially perpendicular to each other. In this case, the polarization directions of the light of the first central wavelength and the light of the second central wavelength incident on each pixel of the image sensor are clearly separated, improving the accuracy of the luminance calculation of the bright pupil image and the dark pupil image, and as a result, improving the accuracy of the detection of the position of the pupil image.

[0063] In addition, in the image sensor, a polarizer that transmits linearly polarized light of four different angles may be attached to every four adjacent pixels, and the calculation device may calculate the luminance corresponding to each of the light of the first and second central wavelengths based on the luminance of at least two of the four adjacent pixels in the eye image. In this case, it is possible to uniformize the arrangement balance of a plurality of pixels that detect various polarization direction components of the light of the first and second central wavelengths on the image sensor. This makes it possible to uniformize the images of the bright pupil image and the dark pupil image, and stabilize the accuracy of the position of the detected pupil image.

[0064] The first light source may be arranged at a first distance from the center of the aperture of the camera, and the second light source may be arranged at a second distance from the center of the aperture that is greater than the first distance. In this case, the luminance difference between the image of the pupil illuminated with the light of the first central wavelength and the image of the pupil illuminated with the light of the second central wavelength can be made larger. As a result, the accuracy of detecting the position of the pupil image using the comparison image can be further improved.

[0065] The second splitting element may be disposed inside the first splitting element, which can prevent uneven brightness of the pupil image and its surroundings in the bright pupil image and the dark pupil image, and can make only the pupil image stand out in the comparison image, thereby improving the accuracy of detecting the pupil image position using the comparison image.

[0066] The second dividing element may have a circular shape and be located at the center of the aperture, and the first dividing element may have a ring shape and be located outside the second dividing element. In this case, it is possible to prevent unevenness in the luminance of the pupil image and its surroundings in the bright pupil image and the dark pupil image, and to highlight the luminance difference in the pupil image. As a result, it is possible to make only the pupil image more prominent in the comparison image, and to further improve the detection accuracy of the pupil image position using the comparison image.

[0067] The first and second splitting elements may be arranged on both sides of a line on the optical element, and the first and second light sources may be arranged to sandwich the opening from both sides. In this case, unevenness in the luminance of the pupil image and its surroundings in the bright pupil image and the dark pupil image can be prevented, only the pupil image can be made to stand out in the comparison image, and the detection accuracy of the pupil image position using the comparison image can be improved.

[0068] Furthermore, the first dividing element may have two or more dividing shapes of a predetermined shape provided on both sides of a line on the optical element, and the second dividing element may be disposed between the two or more dividing shapes. In this case, it is possible to prevent unevenness in the luminance of the pupil image and its surroundings in the bright pupil image and the dark pupil image, and to highlight the luminance difference in the pupil image. As a result, it is possible to make only the pupil image more noticeable in the comparison image, and to further improve the detection accuracy of the pupil image position using the comparison image.

[0069] The optical element may have a plurality of division sections that are divided into a plurality of parts in a predetermined shape, and the first division element and the second division element may be assigned to the plurality of division sections. In this case, it is also possible to prevent unevenness in the luminance of the pupil image and its surroundings in the bright pupil image and the dark pupil image, and to highlight the luminance difference in the pupil image. As a result, it is possible to make only the pupil image more noticeable in the comparison image, and to further improve the detection accuracy of the pupil image position using the comparison image.

[0070] Furthermore, the computing device may calculate the position of the corneal reflection of the subject caused by turning on the first light source or the second light source based on the bright pupil image or the dark pupil image. With this configuration, the position of the corneal reflection can be detected stably.

[0071] Furthermore, the computing device may detect the gaze direction of the subject based on the position of the pupil image calculated based on the comparison image and the position of the corneal reflection image calculated based on the bright pupil image or the dark pupil image. In this way, the gaze direction of the subject can be detected stably. [Industrial Applicability]

[0072] One aspect of the present disclosure is to use a pupil detection device that detects pupils from an image of a person, and to easily improve the accuracy of pupil detection regardless of the state of the subject. [Explanation of symbols]

[0073] A...eyeball, 3A, 3B...light source, 1...pupil detection device, 2...camera, 3...illumination device, 4...control device (arithmetic device), 6...imaging element (image sensor), 8...aperture, 9...optical element, 9A, 9B...splitter element, 21...pixel, 22A, 22B, 22C, 22D...polarizer.

Claims

1. a camera that captures an image of a subject's eye to obtain an eye image; a first light source provided outside or inside an opening of the camera and configured to illuminate the pupil of the subject with light of a first central wavelength with respect to the camera; a second light source provided outside or inside the opening of the camera and illuminating the pupil of the subject with a second central wavelength different from the first central wavelength with respect to the camera; a computing device for processing the eye image, The camera includes: An image sensor in which a polarizer that transmits linearly polarized light having at least two different angles is attached to each of adjacent pixels; an optical element in which a first dividing element is combined with a bandpass filter that passes light of the first central wavelength and a first polarizer that transmits linearly polarized light at a predetermined angle, and a second dividing element is combined with a bandpass filter that passes light of the second central wavelength and a second polarizer that transmits linearly polarized light at an angle different from the predetermined angle, are provided along the opening between the opening and the image sensor; having The computing device includes: calculating a luminance corresponding to light of the first central wavelength based on the luminance of the adjacent pixels in the eye image, and combining the calculated luminances to obtain a bright pupil image in which the pupil of the subject is relatively bright; calculating a luminance corresponding to light of the second central wavelength based on the luminance of the adjacent pixels in the eye image, and combining the calculated luminances to obtain a dark pupil image in which the pupil of the subject is relatively dark; calculating a position of the pupil image of the subject based on a comparison image obtained by comparing the bright pupil image with the dark pupil image; Pupil detection device.

2. The polarization direction of light transmitted through the first polarizer and the polarization direction of light transmitted through the second polarizer are substantially perpendicular to each other.

2. The pupil detection device according to claim 1.

3. In the image sensor, a polarizer that transmits linearly polarized light having four different angles is attached to every four adjacent pixels, The computing device includes: calculating luminances corresponding to the light of the first and second central wavelengths based on luminances of at least two pixels among the four adjacent pixels in the eye image; 3. The pupil detection device according to claim 1 or 2.

4. the first light source is disposed at a first distance from a center of an aperture of the camera; The second light source is disposed at a second distance from the center of the opening that is greater than the first distance. The pupil detection device according to any one of claims 1 to 3.

5. The second dividing element is disposed inside the first dividing element. The pupil detection device according to any one of claims 1 to 4.

6. the second dividing element has a circular shape disposed at the center of the opening; The first divided element has a ring shape disposed outside the second divided element.

6. The pupil detection device according to claim 5.

7. the first and second dividing elements are configured to be disposed on opposite sides of a line on the optical element; The first light source and the second light source are arranged to sandwich the opening from both sides. The pupil detection device according to any one of claims 1 to 4.

8. the first dividing element has two or more dividing shapes of a predetermined shape provided on both sides of a line on the optical element; The second divided element is disposed between the two or more divided shapes.

8. A pupil detection device according to claim 7.

9. The optical element has a plurality of divided portions that are divided into a plurality of portions in a predetermined shape, The first division element and the second division element are assigned to the plurality of division parts. The pupil detection device according to any one of claims 1 to 4.

10. The arithmetic device calculates a position of a corneal reflection image of the subject caused by turning on the first light source or the second light source based on the bright pupil image or the dark pupil image. The pupil detection device according to any one of claims 1 to 9.

11. the arithmetic device detects the gaze direction of the subject based on the position of the pupil image calculated based on the comparison image and the position of the corneal reflection image calculated based on the bright pupil image or the dark pupil image. The pupil detection device according to claim 10.

Images