Distance calculation device, distance calculation method, and distance calculation program
The distance calculation device uses two light sources to simplify the detection of the imaging unit to subject distance by calculating corneal reflection centers, providing accurate distance measurement without complex configurations.
Patent Information
- Application Number
- JP2020139307
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-08-20
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-08-20
AI Technical Summary
Existing line of sight detection devices require complex configurations for calculating the distance between an imaging unit and a subject, such as using ultrasonic sensors or stereo cameras.
A distance calculation device utilizing two light sources positioned differently to irradiate a subject's eyeball, capturing images to determine corneal reflection centers, and calculating the center-to-center distance and corneal radius to determine the target distance.
Enables distance calculation with a simple configuration, allowing for accurate and efficient detection of the imaging unit to subject distance.
Smart Images

Figure 0007593008000001 
Figure 0007593008000002 
Figure 0007593008000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a distance calculation device, a distance calculation method, and a distance calculation program. [Background technology]
[0002] A gaze detection device is known that emits detection light from multiple light sources and irradiates the eyeball of a subject, acquires an image of the eyeball irradiated with the detection light using a single imaging unit, and detects the gaze direction of the subject based on the acquired image and the distance between the imaging unit and the subject (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 4649319 Summary of the Invention [Problem to be solved by the invention]
[0004] In the above-mentioned gaze detection device, in order to detect the distance between the imaging unit and the subject, a method using an ultrasonic sensor, a method using a stereo camera, a method obtaining the distance from the lens position focused by the camera, etc. However, each of these methods results in a complicated configuration.
[0005] The present invention has been made in consideration of the above, and has an object to provide a distance calculation device, a distance calculation method, and a distance calculation program that are capable of calculating the distance between an imaging unit and a subject with a simple configuration. [Means for solving the problem]
[0006] The distance calculation device of the present invention includes a first light source and a second light source that emit detection light from different positions and irradiate it onto at least one of the subject's eyeballs, an imaging unit that captures an image of the subject's eyeball irradiated with the detection light, a position calculation unit that calculates the position of a first corneal reflection center due to the detection light from the first light source and the position of a second corneal reflection center due to the detection light from the second light source based on the image of the subject's eyeball captured by the imaging unit, a center-to-center distance calculation unit that calculates the center-to-center distance between the position of the first corneal reflection center and the second corneal reflection center, and a target distance calculation unit that calculates the target distance between the imaging unit and the subject's eyeball based on the center-to-center distance and the corneal curvature radius of the subject's eyeball.
[0007] A distance calculation method according to the present invention includes emitting detection light from a first light source and a second light source arranged at different positions and irradiating the detection light onto at least one of the subject's eyes, capturing an image of the subject's eyeball irradiated with the detection light using an imaging unit, calculating the position of a first corneal reflection center due to the detection light from the first light source and the position of a second corneal reflection center due to the detection light from the second light source based on the captured image of the subject's eyeball, calculating the center-to-center distance between the position of the first corneal reflection center and the second corneal reflection center, and calculating a target distance between the imaging unit and the subject's eyeball based on the center-to-center distance and a corneal curvature radius of the subject's eyeball.
[0008] The distance calculation program of the present invention causes a computer to execute the following processes: emitting detection light from a first light source and a second light source arranged at different positions and irradiating the detection light onto at least one of the subject's eyes; capturing an image of the subject's eyeball irradiated with the detection light using an imaging unit; calculating the position of a first corneal reflection center due to the detection light from the first light source and the position of a second corneal reflection center due to the detection light from the second light source based on the captured image of the subject's eyeball; calculating the center-to-center distance between the position of the first corneal reflection center and the second corneal reflection center; and calculating the target distance between the imaging unit and the subject's eyeball based on the center-to-center distance and the corneal curvature radius of the subject's eyeball. Effect of the Invention
[0009] According to the present invention, it is possible to calculate the distance between the imaging unit and the subject with a simple configuration. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a gaze detection device according to the present embodiment. [Diagram 2] FIG. 2 is a functional block diagram illustrating an example of the gaze detection device. [Diagram 3] FIG. 3 is a diagram illustrating a state in which detection light from a light source unit is irradiated onto the eyeball of a subject. [Figure 4] FIG. 4 is a diagram illustrating a state in which detection light from a light source unit is irradiated onto the eyeball of a subject. [Diagram 5] FIG. 5 is a diagram illustrating a state in which detection light from a light source unit is irradiated onto the eyeball of a subject. [Figure 6] FIG. 6 is a diagram showing an example of the positional relationship between the light source unit, the imaging unit, and the eyeball of the subject. [Figure 7] FIG. 7 is a diagram showing an example of image data of an eyeball captured by the imaging unit. [Figure 8] FIG. 8 is a graph showing an example of the relationship between the center distance and the target distance. [Figure 9] FIG. 9 is a schematic diagram for explaining the principle of the calibration process according to this embodiment. [Figure 10] FIG. 10 is a schematic diagram for explaining the principle of the gaze detection process according to this embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of a distance calculation method according to this embodiment. [Figure 12] FIG. 12 is a diagram showing an example of image data of an eyeball captured in a state in which the first light source and the second light source are turned on at the same timing. [Figure 13] FIG. 13 is a diagram showing an example of image data of an eyeball captured in a state in which the first light source and the second light source are turned on at the same timing. [Figure 14] FIG. 14 is a diagram showing an example of image data of an eyeball captured in a state in which the first light source and the second light source are turned on at the same timing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, the components in the following embodiments include those that are replaceable and easy for a person skilled in the art, or those that are substantially the same.
[0012] In the following description, a three-dimensional global coordinate system is set to explain the positional relationship of each part. The direction parallel to the first axis of the specified plane is the X-axis direction, the direction parallel to the second axis of the specified plane perpendicular to the first axis is the Y-axis direction, and the direction parallel to the third axis perpendicular to both the first and second axes is the Z-axis direction. The specified plane includes the XY plane.
[0013] [Distance calculation device, line of sight detection device] 1 is a diagram showing an example of a gaze detection device 100 according to the present embodiment. The gaze detection device 100 according to the present embodiment detects the gaze of a subject and outputs the detection result. The gaze detection device 100 detects the gaze based on, for example, the position of the subject's pupil and the position of the corneal reflection.
[0014] 1, the gaze detection device 100 includes a display device 10, an image capture device 20, a computer system 30, an output device 40, an input device 50, and an input / output interface device 60. The display device 10, the image capture device 20, the computer system 30, the output device 40, and the input device 50 perform data communication via the input / output interface device 60. The display device 10 and the image capture device 20 each have a drive circuit (not shown).
[0015] The display device 10 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). In this embodiment, the display device 10 has a display unit 11. The display unit 11 displays information such as an image. The display unit 11 is substantially parallel to an XY plane. The X-axis direction is the left-right direction of the display unit 11, the Y-axis direction is the up-down direction of the display unit 11, and the Z-axis direction is the depth direction perpendicular to the display unit 11. The display device 10 may be a head-mounted display device. In the case of a head-mounted display, a configuration such as the image acquisition device 20 is disposed in a head-mounted module.
[0016] The image acquisition device 20 acquires image data of the left and right eyes EB of the subject, and transmits the acquired image data to the computer system 30. The image acquisition device 20 includes a light source unit 21 and an imaging unit 22.
[0017] The light source unit 21 emits detection light to the eye EB of the subject. The light source unit 21 has a first light source 21A and a second light source 21B arranged at different positions. The first light source 21A and the second light source 21B include an LED (light emitting diode) light source and can emit near-infrared light with a wavelength of, for example, 850 [nm]. The first light source 21A and the second light source 21B can individually control the timing of emitting the detection light.
[0018] The imaging unit 22 generates image data by capturing images of the left and right eyeballs EB of the subject. In this embodiment, one imaging unit 22 is provided. As the imaging unit 22, various cameras are used according to a method for detecting the line of sight of the subject. In the case of a method for detecting the line of sight based on the position of the subject's pupil and the position of the corneal reflection as in this embodiment, the imaging unit 22 has an infrared camera, and has an optical system capable of transmitting near-infrared light with a wavelength of 850 [nm], for example, and an imaging element capable of receiving the near-infrared light. The imaging unit 22 outputs a frame synchronization signal. The period of the frame synchronization signal can be, for example, 20 [msec], but is not limited to this. In this embodiment, the imaging unit 22 is one camera. The imaging unit 22 is disposed at a position intermediate between the first light source 21A and the second light source 21B. The image data acquired by the imaging unit 22 is configured such that pixels having a luminance set by, for example, an 8-bit value (0 to 255) are two-dimensionally arranged. The smaller the luminance value, the darker the image, and the larger the luminance value, the brighter the image. A pixel with a luminance value of 0 will be displayed as black in the image data. A pixel with a luminance value of 255 will be displayed as white in the image data.
[0019] The computer system 30 controls the overall operation of the gaze detection device 100. The computer system 30 includes an arithmetic processing device 30A and a storage device 30B. The arithmetic processing device 30A includes a microprocessor such as a central processing unit (CPU). The storage device 30B includes a memory or storage such as a read only memory (ROM) and a random access memory (RAM). The arithmetic processing device 30A performs arithmetic processing according to a computer program 30C stored in the storage device 30B.
[0020] The output device 40 includes a display device such as a flat panel display. The output device 40 may also include a printing device. The input device 50 generates input data when operated. The input device 50 includes a keyboard or a mouse for a computer system. The input device 50 may also include a touch sensor provided on a display unit of the output device 40, which is a display device.
[0021] In the gaze detection device 100 according to the present embodiment, the display device 10 and the computer system 30 are separate devices. The display device 10 and the computer system 30 may be integrated. For example, the gaze detection device 100 may include a tablet-type personal computer. In this case, the tablet-type personal computer may be equipped with a display device, an image acquisition device, a computer system, an input device, an output device, and the like.
[0022] FIG. 2 is a functional block diagram showing an example of the gaze detection device 100. As shown in FIG. 2, the computer system 30 has an imaging control unit 31, a position calculation unit 32, a center-to-center distance calculation unit 33, a target distance calculation unit 34, a detection processing unit 35, and a storage unit 36. The functions of the computer system 30 are realized by an arithmetic processing device 30A and a storage device 30B (see FIG. 1). Note that some functions of the computer system 30 may be provided outside the gaze detection device 100. In this embodiment, the light source unit 21 (the first light source 21A and the second light source 21B) and the imaging unit 22 of the image acquisition device 20, the imaging control unit 31, the position calculation unit 32, the center-to-center distance calculation unit 33, and the target distance calculation unit 34 constitute a distance calculation device 80. Note that the distance calculation device 80 may include at least one of the detection processing unit 35 and the storage unit 36.
[0023] The imaging control unit 31 controls the light source unit 21 and the imaging unit 22. The imaging control unit 31 controls the timing and emission time of the detection light for each of the first light source 21A and the second light source 21B. The imaging control unit 31 controls the imaging timing for the imaging unit 22. The imaging control unit 31 causes the first light source 21A and the second light source 21B to emit the detection light in synchronization with the period of the frame synchronization signal of the imaging unit 22. In this embodiment, the imaging control unit 31 causes the first light source 21A and the second light source 21B to alternately emit the detection light for each period of the frame synchronization signal, for example. The imaging control unit 31 acquires image data acquired by the image acquisition device 20. The imaging control unit 31 stores the acquired image data in the storage unit 36.
[0024] The position calculation unit 32 calculates the position of a first corneal reflection center 121 (see FIG. 1, FIG. 3, etc.) by the detection light from the first light source 21A and the position of a second corneal reflection center 122 (see FIG. 1, FIG. 3, etc.) by the detection light from the second light source 21B based on image data of the subject's eyeball captured by the imaging unit 22. The first corneal reflection center 121 and the second corneal reflection center 122 are the centers of the corneal reflection images (first corneal reflection image 120A, second corneal reflection image 120B) by the respective detection lights. The position calculation unit 32 stores the calculated positions of the first corneal reflection center 121 and the second corneal reflection center 122 in the memory unit 36.
[0025] The center-to-center distance calculation unit 33 calculates the center-to-center distance between the position of the first corneal reflection center 121 and the second corneal reflection center 122 .
[0026] The object distance calculation unit 34 calculates the object distance between the imaging unit 22 and the subject's eyeball EB based on the center-to-center distance and the corneal curvature radius of the subject's eyeball. The object distance calculation unit 34 stores the calculated object distance in the memory unit 36. The object distance calculation unit 34 can calculate the object distance by using the corneal curvature radius of the subject's eyeball as, for example, a predetermined constant. The corneal curvature radius is the distance between the surface of the cornea of the eyeball EB and the corneal curvature center. The corneal curvature radius and the corneal curvature center will be described later.
[0027] The detection processing unit 35 calculates the position of the center of corneal curvature of the subject's eyeball based on the image data captured by the imaging unit 22 and the target distance. The detection processing unit 35 detects the position of the center of the subject's pupil based on the image data. The pupil center is the center of the pupil 112. The detection processing unit 35 calculates the position of the center of corneal curvature of the eyeball EB from the calculated position of the pupil center and the positions of the first corneal reflection center 121 and the second corneal reflection center 122 calculated by the position calculation unit 32.
[0028] An overview of the process of calculating the position of the corneal curvature center in the detection processing unit 35 will be described. In this embodiment, a case will be described in which the eyeball EB is illuminated by the first light source 21A and the second light source 21B, and the eyeball EB is imaged by one imaging unit 22. Note that the present invention is not limited to a case in which there are two light sources and one imaging unit, and a similar explanation can be given for a case in which there is one light source and two cameras.
[0029] 3 is a diagram illustrating a state in which the detection light from the light source unit 21 is irradiated onto the eye EB of the subject. As illustrated in FIG. 3, in this embodiment, the first light source 21A and the second light source 21B of the light source unit 21 are disposed at positions symmetrical to the left and right with respect to the imaging unit 22. Therefore, in this case, it can be considered that a virtual light source (reference position of the light source) 21C exists at the position of the imaging unit 22.
[0030] The first corneal reflection center 121 indicates the corneal reflection center in an image when the eye EB is irradiated with detection light from the first light source 21A. The second corneal reflection center 122 indicates the corneal reflection center in an image when the eye EB is irradiated with detection light from the second light source 21B. The corneal reflection center 124 indicates the corneal reflection center corresponding to the virtual light source 21C.
[0031] The position of the corneal reflection center 124 is calculated based on the positions of the first corneal reflection center 121 and the second corneal reflection center 122. The detection processing unit 35 calculates the positions of the first corneal reflection center 121 and the second corneal reflection center 122 on the image of the imaging unit 22. The detection processing unit 35 converts the positions into a three-dimensional coordinate system based on the object distance between the imaging unit 22 and the subject's eyeball EB. The conversion parameters for converting the positions into the three-dimensional coordinate system are stored in, for example, the storage unit 36. The detection processing unit 35 calculates the position of the corneal reflection center 124 in the three-dimensional coordinate system based on the positions of the first corneal reflection center 121 and the second corneal reflection center 122 defined in the three-dimensional coordinate system.
[0032] Corneal curvature center 110 exists on straight line 123 connecting virtual light source 21C and corneal reflection center 124. Detection processing unit 35 calculates the position on straight line 123 at a predetermined distance from corneal reflection center 124 as the position of corneal curvature center 110. Corneal curvature radius r is used as the predetermined value. Corneal curvature radius r is the distance between the corneal surface and corneal curvature center 110. As the value of corneal curvature radius r, a constant that is predetermined based on, for example, a general corneal curvature radius value can be used. This value of corneal curvature radius r is also used in target distance calculation unit 34 described above, for example.
[0033] The detection processor 35 detects the gaze vectors of the left and right eyes EB of the subject based on the positions of the pupil center and the corneal curvature center. After detecting the gaze vectors, the detection processor 35 detects the position of the gaze point indicating the intersection of the gaze vector and the display unit 11.
[0034] The storage unit 36 stores various data and programs for performing processing in each part of the computer system 30. The storage unit 36 stores, for example, data on an image to be displayed on the display unit 11. The storage unit 36 stores image data acquired by the imaging control unit 31, the positions of the first corneal reflection center 121 and the second corneal reflection center 122 calculated by the position calculation unit 32, the object distance calculated by the object distance calculation unit 34, and the corneal curvature radius r calculated by the detection processing unit 35.
[0035] The memory unit 36 also stores a distance calculation program that causes the computer to execute the following processes: emitting detection light from the first light source 21A and the second light source 21B, which are arranged at different positions, and irradiating it onto at least one of the subject's eyes EB; capturing an image of the subject's eye EB irradiated with the detection light using the imaging unit 22; calculating the position of the first corneal reflection center 121 based on the detection light from the first light source 21A and the position of the second corneal reflection center 122 based on the detection light from the second light source 21B based on the captured image of the subject's eye; calculating the center-to-center distance between the position of the first corneal reflection center 121 and the second corneal reflection center 122; and calculating the target distance between the imaging unit 22 and the subject's eye EB based on the center-to-center distance and the corneal curvature radius r of the subject's eye.
[0036] [Distance calculation method] Next, a distance calculation method according to the present embodiment will be described. Fig. 4 and Fig. 5 are diagrams that show a state in which the detection light from the light source unit 21 is irradiated onto the subject's eye EB. Hereinafter, the subject's right eye EB (right eye ER) will be described, but the same explanation can be applied to the left eye EB (left eye EL). As shown in Figs. 4 and 5, when the detection light is irradiated onto the eye EB from the first light source 21A and the second light source 21B, a corneal reflection image is formed on the cornea of the eye EB. Here, the center of the corneal reflection image of the first light source 21A is the first corneal reflection center 121. Also, the center of the corneal reflection image of the second light source 21B is the second corneal reflection center 122.
[0037] 5, as the distance between the imaging unit 22 and the subject changes, the center-to-center distance d between the first corneal reflection center 121 and the second corneal reflection center 122 changes. For example, d1>d2 holds between the center-to-center distance d1 when the object distance, which is the distance between the imaging unit 22 and the subject's eyeball EB, is x1 (position P1) and the center-to-center distance d2 when the object distance is x2 larger than x1 (position P2).
[0038] FIG. 6 is a diagram showing an example of the positional relationship among a light source unit, an imaging unit, and the eye of a subject. As shown in FIG. 6, let the distance between the first light source 21A and the imaging unit 22 be a, the distance between the second light source 21B and the imaging unit 22 be b, the object distance between the imaging unit 22 and the subject be x, the distance between the corneal reflection center 121 and the corneal reflection center 122 detected on the eyeball EB be d, and the corneal radius of curvature be r. In this example, the object distance x is the distance between the imaging unit 22 and the corneal curvature center 110 of the subject's eyeball EB. Here, since r << x, it can be approximated that the angle formed by the straight line L1 passing through the corneal curvature center 110 and the first light source 21A and the straight line L2 passing through the corneal curvature center 110 and the corneal reflection center 121 is equal to the angle formed by the straight line L3 passing through the corneal curvature center 110 and the imaging unit 22 and the straight line L2. Hereinafter, this angle is denoted as θ1. Similarly, it can be approximated that the angle formed by the straight line L4 passing through the corneal curvature center 110 and the second light source 21B and the straight line L5 passing through the corneal curvature center 110 and the corneal reflection center 121 is equal to the angle formed by the straight line L3 passing through the corneal curvature center 110 and the imaging unit 22 and the straight line L5. Hereinafter, this angle is denoted as θ2.
[0039] In this case, the following equations 1 to 3 hold. θ1 = arctan(a / x) / 2 ···(Equation 1) θ2 = arctan(b / x) / 2 ···(Equation 2) d = r·sinθ1 + r·sinθ2 ···(Equation 3)
[0040] 7 is a diagram showing an example of image data of the eye EB captured by the imaging unit 22. As shown in FIG. 7, in this embodiment, image data IM1 captured in a state where the eye EB is irradiated with detection light from the first light source 21A and image data IM2 captured in a state where the eye EB is irradiated with detection light from the second light source 21B are acquired as separate image data. The image data IM1 includes a corneal reflection image (hereinafter, referred to as a first corneal reflection image 120A) due to the detection light from the first light source 21A. The image data IM2 includes a corneal reflection image (hereinafter, referred to as a second corneal reflection image 120B) due to the detection light from the second light source 21B. In this embodiment, the first light source 21A and the second light source 21B are turned on at different timings. Therefore, the first corneal reflection image 120A and the second corneal reflection image 120B are reflected in one image data IM1 only the first corneal reflection image 120A, and in the other image data IM2 only the second corneal reflection image 120B.
[0041] The position calculation unit 32 calculates the position of the first corneal reflection center 121 by the detection light from the first light source 21A based on the acquired image data IM1. The position calculation unit 32 also calculates the position of the second corneal reflection center 122 by the detection light from the second light source 21B based on the acquired image data IM2. In the image data IM1 and IM2, the pixels in which the first corneal reflection image 120A and the second corneal reflection image 120B are formed have higher brightness than other pixels. The position calculation unit 32 searches for the pixel with the highest brightness in the image data IM1 and IM2, obtains a high brightness area in which pixels having brightness within a certain range exist based on the pixel, obtains the brightness center of gravity of this area, and sets the coordinates of the brightness center of gravity on the image as the first corneal reflection center 121 and the second corneal reflection center 122. Note that in FIG. 7, the brightness of the high brightness area is shown to be uniform, but there may be cases where the brightness decreases from one pixel in the high brightness area to the surrounding pixels.
[0042] The center distance calculation unit 33 calculates the actual distance d (mm) between the first corneal reflection center 121 and the second corneal reflection center 122 from the distance (number of pixels of the imaging element) d' (pixels) between the first corneal reflection center 121 included in the image data IM1 and the second corneal reflection center 122 included in the image data IM2. Note that in Fig. 7, the first corneal reflection center 121 and the second corneal reflection center 122 are shown as black dots to make the positions easier to distinguish, but in reality there are no black dots.
[0043] Here, if the focal length of the lens constituting the imaging unit 22 is f, and the distance (pitch) between pixels of the imaging element constituting the imaging unit 22 is p, then the following formula 4 is established.
[0044] d´=(f d / x) / p (Equation 4)
[0045] Solving equations 1 to 4 above for d' gives us d´=r·(sin(arctan(a / x) / 2) +sin(arctan(b / x) / 2)) f / (p x) (Equation 5) It becomes.
[0046] In Equation 5, for example, if a, b = 100 (mm), f = 12 (mm), r = 7.9 (mm), and p = 0.0048 (mm / pixel), the relation between x and d' is obtained as shown in Equation 6 below. In this embodiment, the value of the corneal curvature radius r (7.9) is set as a predetermined constant. For the value of this corneal curvature radius r, a value determined in advance from, for example, a general corneal curvature radius value, or a value calculated in a calibration process may be used.
[0047] d´=7.9·(2·sin(arctan(100 / x) / 2))·12 / (0.0048 x) (Equation 6)
[0048] By using the above formula 6, the object distance x can be calculated based on the center-to-center distance d'. Fig. 8 is a graph showing an example of the relationship between the center-to-center distance d' and the object distance x. Fig. 8 shows that, regarding the relationship between the center-to-center distance d' and the object distance x shown in formula 6, the center-to-center distance d' becomes smaller as the object distance x becomes larger.
[0049] [Gaze detection method] Next, an example of a gaze detection method using the gaze detection device 100 according to this embodiment will be described. In the gaze detection method according to this embodiment, a calibration process is performed, and then a gaze point detection process is performed. First, the principle of the calibration process will be described. FIG. 9 is a schematic diagram for explaining the principle of the calibration process according to this embodiment. In the calibration process, a target position 130 is set so that the subject will gaze at it. The target position 130 is defined in a three-dimensional global coordinate system. In this case, the detection processing unit 35 displays a target image at the set target position 130.
[0050] The first light source 21A and the second light source 21B alternately illuminate the eye EB with detection light. The imaging unit 22 images the eye EB illuminated with the detection light. The detection processing unit 35 detects the position of the pupil center 112C and the position of the corneal reflection center 113C corresponding to the virtual light source 21C based on the image data of the eye EB. The detection processing unit 35 converts the positions of the pupil center 112C and the corneal reflection center 113C into the global coordinate system using the object distance x calculated by the object distance calculation unit 34.
[0051] The detection processing unit 35 calculates the position of the corneal curvature center 110 based on the position of the virtual light source 21C, the position of the target position 130, the position of the pupil center 112C, and the position of the corneal reflection center 113C. Specifically, the detection processing unit 35 obtains a first straight line 141 connecting the virtual light source 21C and the corneal reflection center 113C. The detection processing unit 35 also obtains a second straight line 142 connecting the target position 130 and the pupil center 112C. The detection processing unit 35 obtains the intersection of the first straight line 141 and the second straight line 142 as the position of the corneal curvature center 110. Then, the detection processing unit 35 calculates the distance between the corneal curvature center 110 and the corneal reflection center 113C as the corneal curvature radius r. The detection processing unit 35 stores the calculated value of the corneal curvature radius r in the storage unit 36 as calibration data.
[0052] Next, the principle of the gaze detection process will be described. Fig. 10 is a schematic diagram for explaining the principle of the gaze detection process according to this embodiment. In the gaze detection process, similar to the calibration process, the first light source 21A and the second light source 21B alternately emit detection light to illuminate the eyeball EB, and the image capturing unit 22 captures the image of the eyeball EB. The detection processing unit 35 detects the position of the pupil center 112C and the position of the corneal reflection center 113C based on the acquired image data of the eyeball EB. The detection processing unit 35 converts the positions of the pupil center 112C and the corneal reflection center 113C into the global coordinate system using the target distance x calculated by the target distance calculation unit 34.
[0053] The detection processing unit 35 calculates the position of the corneal curvature center 110 based on the positions of the virtual light source 21C, the pupil center 112C, the corneal reflection center 113C, and the corneal curvature radius r calculated in the calibration process. Specifically, the detection processing unit 35 obtains a straight line 173 connecting the virtual light source 21C and the corneal reflection center 113C. The detection processing unit 35 also obtains a position away from the corneal reflection center 113C by a distance equivalent to the corneal curvature radius r toward the inside of the eyeball EB as the position of the corneal curvature center 110. The detection processing unit 35 obtains a straight line 178 connecting the pupil center 112C and the corneal curvature center 110, and calculates the position of the intersection 166 between the straight line 178 and the display unit 11 as the position of the gaze point.
[0054] Next, an example of a distance calculation method according to this embodiment will be described with reference to FIG. 11. FIG. 11 is a flowchart showing an example of a distance calculation method according to this embodiment. As shown in FIG. 11, under the control of the imaging control unit 31, one of the first light source 21A and the second light source 21B is caused to emit light to irradiate the eye EB with detection light (step S101), and the imaging unit 22 captures an image of the eye EB of the subject (step S102). Also, under the control of the imaging control unit 31, the other of the first light source 21A and the second light source 21B is caused to emit light to irradiate the eye EB with detection light (step S103), and the imaging unit 22 captures an image of the eye EB of the subject (step S104). The imaging control unit 31 acquires image data of the left and right eye EB captured by the imaging unit 22.
[0055] The position calculation unit 32 determines the first corneal reflection center 121 in the first corneal reflection image 120A and the second corneal reflection center 122 in the second corneal reflection image 120 for each of the left and right eyes EB based on the acquired image data (step S105). The center-to-center distance calculation unit 33 calculates the center-to-center distance d' between the first corneal reflection center 121 and the second corneal reflection center 122 (step S106). The object distance calculation unit 34 calculates the object distance x based on the calculated center-to-center distance d' (step S107).
[0056] As described above, the distance calculation device 80 of this embodiment includes a first light source 21A and a second light source 21B that emit detection light from different positions and irradiate it onto at least one of the subject's eyes EB, an imaging unit 22 that captures an image of the subject's eye EB irradiated with the detection light, a position calculation unit 32 that calculates the position of the first corneal reflection center 121 based on the detection light from the first light source 21A and the position of the second corneal reflection center 122 based on the detection light from the second light source 21B based on the image of the subject's eye EB captured by the imaging unit 22, a center-to-center distance calculation unit 33 that calculates the center-to-center distance d' between the position of the first corneal reflection center 121 and the second corneal reflection center 122, and a target distance calculation unit 34 that calculates the target distance between the imaging unit 22 and the subject's eye EB based on the center-to-center distance d' and the corneal curvature radius r of the subject's eye EB.
[0057] The distance calculation method of this embodiment includes emitting detection light from a first light source 21A and a second light source 21B arranged at different positions and irradiating it onto at least one of the subject's eyes EB, capturing an image of the subject's eye EB irradiated with the detection light using an imaging unit 22, calculating the position of a first corneal reflection center 121 based on the detection light from the first light source 21A and the position of a second corneal reflection center 122 based on the detection light from the second light source 21B based on the captured image of the subject's eye EB, calculating a center-to-center distance d' between the position of the first corneal reflection center 121 and the second corneal reflection center 122, and calculating a target distance between the imaging unit 22 and the subject's eye EB based on the center-to-center distance d' and the corneal curvature radius r of the subject's eye EB.
[0058] The distance calculation program of this embodiment causes a computer to execute the following processes: emitting detection light from a first light source 21A and a second light source 21B arranged at different positions and irradiating it onto at least one of the subject's eyes EB; capturing an image of the subject's eye EB irradiated with the detection light using the imaging unit 22; calculating the position of a first corneal reflection center 121 based on the detection light from the first light source 21A and the position of a second corneal reflection center 122 based on the detection light from the second light source 21B based on the captured image of the subject's eye EB; calculating a center-to-center distance d' between the position of the first corneal reflection center 121 and the second corneal reflection center 122; and calculating a target distance between the imaging unit 22 and the subject's eye EB based on the center-to-center distance d' and the corneal curvature radius r of the subject's eye EB.
[0059] According to the above configuration, the center-to-center distance d' between the position of the first corneal reflection center 121 and the second corneal reflection center 122 is calculated based on image data from the imaging unit 22, and the object distance between the imaging unit 22 and the subject's eyeball EB is calculated based on the calculated center-to-center distance d' and the corneal curvature radius r of the subject's eyeball EB, making it possible to calculate the distance between the imaging unit 22 and the subject with a simple configuration without using a complex configuration such as an ultrasonic sensor or stereo camera.
[0060] In the distance calculation device 80 according to this embodiment, the first light source 21A and the second light source 21B alternately emit detection light, and the position calculation unit 32 calculates the position of the first corneal reflection center 121 based on an image captured at the timing when the detection light is emitted from the first light source 21A, and calculates the position of the second corneal reflection center 122 based on an image captured at the timing when the detection light is emitted from the second light source 21B. When the detection light is emitted simultaneously from the first light source 21A and the second light source 21B, the first corneal reflection image 120A by the detection light from the first light source 21A and the second corneal reflection image 120B by the detection light from the second light source 21B may overlap, and the detection accuracy of the first corneal reflection center 121 and the second corneal reflection center 122 may decrease. In contrast to this, in the above configuration, the first corneal reflection image 120A and the second corneal reflection image 120B can be acquired as separate images, so that the first corneal reflection center 121 and the second corneal reflection center 122 can be detected with high accuracy.
[0061] In the distance calculation device 80 according to the present embodiment, the target distance calculation unit 34 calculates the target distance x by setting the corneal curvature radius r of the subject's eyeball EB as a predetermined constant. In this configuration, the target distance x can be easily calculated by setting the corneal curvature radius r of the subject's eyeball EB as a predetermined constant.
[0062] The technical scope of the present invention is not limited to the above-mentioned embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. For example, in the above-mentioned embodiment, the first light source 21A and the second light source 21B are turned on at different timings, but the present invention is not limited to this. The first light source 21A and the second light source 21B may be turned on at the same timing and captured by the imaging unit 22.
[0063] 12 to 14 are diagrams showing examples of image data of the eye EB captured in a state where the first light source 21A and the second light source 21B are turned on at the same timing. FIG. 12 shows an example in which the object distance x between the imaging unit 22 and the subject is the same as that in FIG. 7. FIG. 13 shows an example in which the object distance is closer than that in the state of FIG. 12. FIG. 14 shows an example in which the object distance is even closer than that in the state of FIG. 13. Note that in FIGS. 12 to 14, the first corneal reflection center 121 and the second corneal reflection center 122 are shown as black dots to make the positions easier to distinguish, but in reality, there are no black dots.
[0064] In the image data IM3 and IM4 shown in Fig. 12 and Fig. 13, the image of the first corneal reflection image 120A and the image of the second corneal reflection image 120B are separated. In this case, the position calculation unit 32 can obtain the image of the first corneal reflection image 120A and the image of the second corneal reflection image 120B as separate high-luminance areas and can independently obtain the coordinates of the luminance centroids of the separate high-luminance areas. This makes it possible to separately calculate the positions of the first corneal reflection center 121 and the second corneal reflection center 122 from one image data IM3 and IM4, and to calculate the center-to-center distances d3 and d4.
[0065] On the other hand, in the image data IM5 shown in Fig. 14, the image of the first corneal reflection image 120A and the image of the second corneal reflection image 120B are partially joined together. In this state, it is difficult for the position calculation unit 32 to determine the image of the first corneal reflection image 120A and the image of the second corneal reflection image 120B as separate high-luminance regions. Therefore, in such a case, by using the image data IM1 and IM2 obtained by capturing an image of the eye EB with the first light source 21A and the second light source 21B turned on at different times, the positions of the first corneal reflection center 121 and the second corneal reflection center 122 can be reliably calculated, and the center-to-center distance d5 can be calculated. [Explanation of symbols]
[0066] EB...eyeball, d, d1, d2, d3, d4, d5...center distance, IM1, IM2, IM3, IM4, IM5...image data, r...corneal curvature radius, x...target distance, 10...display device, 11...display unit, 20...image acquisition device, 21...light source unit, 21A...first light source, 21B...second light source, 21C...virtual light source, 22...imaging unit, 30...computer system, 30A...arithmetic processing unit, 30B...storage device, 30C...computer program, 31...imaging control unit, 32...position position calculation unit, 33...center distance calculation unit, 34...object distance calculation unit, 35...detection processing unit, 36...storage unit, 40...output device, 50...input device, 60...input / output interface device, 80...distance calculation device, 100...gaze detection device, 110...corneal curvature center, 112...pupil, 112C...pupil center, 113C, 121, 122, 124...corneal reflection center, 120A...first corneal reflection image, 120B...second corneal reflection image, 121...first corneal reflection center, 122...second corneal reflection center
Claims
1. a first light source and a second light source that emit detection light from different positions and irradiate at least one eyeball of the subject; an imaging unit that captures an image of the eyeball of the subject irradiated with the detection light; a position calculation unit that calculates a position of a first corneal reflection center due to the detection light from the first light source and a position of a second corneal reflection center due to the detection light from the second light source based on an image of the eyeball of the subject captured by the imaging unit; a center distance calculation unit that calculates a center-to-center distance on an image between a position of the first corneal reflection center and a position of the second corneal reflection center on an image captured by the imaging unit, and calculates an actual center-to-center distance using the calculated center-to-center distance on the image, focal length information of a lens of the imaging unit, and pixel distance information of the imaging unit; an object distance calculation unit that calculates an object distance between the imaging unit and the eyeball of the subject based on the actual center-to-center distance and a corneal curvature radius of the eyeball of the subject; A distance calculation device comprising:
2. the first light source and the second light source alternately emit the detection light, The position calculation unit calculates a position of the first corneal reflection center based on an image captured at a timing when the detection light is emitted from the first light source, and calculates a position of the second corneal reflection center based on an image captured at a timing when the detection light is emitted from the second light source. The distance calculation device according to claim 1 .
3. The object distance calculation unit calculates the object distance by using a corneal curvature radius of the eyeball of the subject as a predetermined constant. The distance calculation device according to claim 1 or 2.
4. emitting detection light from a first light source and a second light source disposed at different positions from each other and irradiating at least one eyeball of the subject; capturing an image of the eyeball of the subject irradiated with the detection light by an imaging unit; calculating a position of a first corneal reflection center due to the detection light from the first light source and a position of a second corneal reflection center due to the detection light from the second light source based on the captured image of the eyeball of the subject; calculating a center-to-center distance on an image between a position of the first corneal reflection center and a position of the second corneal reflection center on an image captured by the imaging unit, and calculating an actual center-to-center distance using the calculated center-to-center distance on the image, focal length information of a lens of the imaging unit, and pixel distance information of the imaging unit; Calculating a target distance between the imaging unit and the eyeball of the subject based on the actual center-to-center distance and a corneal curvature radius of the eyeball of the subject; A distance calculation method including:
5. A process of emitting detection light from a first light source and a second light source disposed at different positions and irradiating the detection light onto at least one eyeball of the subject; A process of capturing an image of the eyeball of the subject irradiated with the detection light by an imaging unit; calculating a position of a first corneal reflection center due to the detection light from the first light source and a position of a second corneal reflection center due to the detection light from the second light source based on an image of the eyeball of the subject; a process of calculating a center-to-center distance on an image between a position of the first corneal reflection center and a position of the second corneal reflection center on an image captured by the imaging unit, and calculating an actual center-to-center distance using the calculated center-to-center distance on the image, focal length information of a lens of the imaging unit, and pixel distance information of the imaging unit; calculating a target distance between the imaging unit and the eyeball of the subject based on the actual center-to-center distance and a corneal curvature radius of the eyeball of the subject; A distance calculation program that causes a computer to execute the above.
Citation Information
Patent Citations
Eye detection system
JP1998073662A
Apparatus, method and program for measuring visual axis
JP2007136000A
Visual line measuring device and visual line measuring program
JP2010259605A
Visual axis controlling device, method of controlling visual axis, and program of the same
JP2011115460A
Line of sight detection support device and line of sight detection support method
JP2015126857A
