Detection device and control method for controlling a detection device

Abstract

The present invention provides a detection device and a control method for controlling the detection device. Gaze detection is performed taking into account the shape of a pupil. A central processing unit (CPU) detects a contour coordinate group of the pupil from an eyeball image obtained as a result of imaging an image of an eyeball of an observer, and performs control to perform gaze detection taking into account the shape of the pupil.

Description

Technical Field

[0001] This invention relates to an optical device including a line-of-sight detection unit. Background Technology

[0002] Traditionally, discussions have focused on so-called line-of-sight detection devices, which detect which position on the observation surface an observer is looking at using optical equipment.

[0003] For example, in Japanese Patent Application Publication No. 61-172552, a parallel beam of light from a light source is projected onto the anterior part of the observer's eyeball, and the visual axis is obtained by using the corneal reflection image of the reflected light from the cornea and the image-forming position of the pupil.

[0004] Japanese Patent 3143490 discusses a method for estimating the pupil circle based on the coordinates of the horizontal / vertical position of an output signal considered to be related to the pupil edge extracted by a region sensor, and applying the center coordinates of the pupil circle to gaze detection. This method estimates the circle at many observation points, thus accurately estimating the original circle even at observation points along, for example, a semicircle or crescent.

[0005] Japanese Patent Application Publication No. 61-172552 and Japanese Patent No. 3143490 are effective when the pupil is perfectly round and without any defects, but they do not consider cases where the pupil shape is elliptical or where part of the pupil's outline is missing. Therefore, errors may occur in the detection results of the pupil center, and the accuracy of gaze detection may be reduced. Summary of the Invention

[0006] The present invention aims to provide a gaze detection device that enables gaze detection taking into account the shape of the pupil, and a control method for controlling the gaze detection device.

[0007] According to one aspect of the invention, the device includes: an imaging unit configured to capture an image of an observer's eyeball; a contour detection unit configured to detect a set of contour coordinates of a pupil from the image of the eyeball acquired from the imaging unit; a calculation unit configured to calculate a pupil center position from the set of contour coordinates; and a gaze detection unit configured to detect the observer's gaze from the pupil center position, wherein the calculation unit is configured to calculate the pupil center position from a second set of contour coordinates obtained by excluding some coordinates from the first set of contour coordinates detected by the detection unit, and the gaze detection unit is configured to detect the observer's gaze based on the pupil center position.

[0008] According to another aspect of the invention, the device includes: a camera unit configured to capture an image of an observer's eyeball; a contour detection unit configured to detect a set of contour coordinates of the pupil from the image of the eyeball acquired from the camera unit; a calculation unit configured to calculate the pupil center position from the set of contour coordinates; and a unit configured to detect the observer's gaze from the pupil center position, wherein the calculation unit is configured to calculate the pupil center position based on the set of contour coordinates and pre-stored information related to the pupil shape.

[0009] According to another aspect of the invention, the device includes: a camera unit configured to capture an image of an observer's eyeball; a unit configured to detect a set of contour coordinates of the pupil from the image of the eyeball acquired from the camera unit; a first calculation unit configured to calculate a first pupil center position from the set of contour coordinates using a circular approximation; a second calculation unit configured to calculate a second pupil center position from the set of contour coordinates using an elliptical approximation; and a gaze detection unit configured to detect the observer's gaze from the pupil center position determined by a judgment unit, and configured to change whether to use the first pupil center position or the second pupil center position to detect the observer's gaze detected by the gaze detection unit based on the absence of pupil contours.

[0010] One aspect of the present invention provides a detection device, comprising: a camera unit configured to capture an image of an observer's eyeball; a detection unit configured to identify a pupil center position based on the outline of the observer's pupil according to the image of the eyeball acquired from the camera unit, and to detect the observer's gaze position using the pupil center position; and a judgment unit configured to determine the degree to which the outline of the pupil is obscured by the observer's eyelid, wherein, if the judgment unit determines that the degree is a first degree, the detection unit uses the pupil center position identified by approximating the outline of the pupil as a first circle to detect the observer's gaze position, and wherein, if the judgment unit determines that the degree is a second degree less than the first degree, the detection unit uses the pupil center position identified by approximating the outline of the pupil as a second circle flatter than the first circle to detect the observer's gaze position.

[0011] Another aspect of the present invention provides a detection device, comprising: a camera unit configured to capture an image of an observer's eyeball; a detection unit configured to identify a pupil center position based on the outline of the observer's pupil according to the image of the eyeball acquired from the camera unit, and to detect the observer's gaze position using the pupil center position; and a determination unit configured to determine whether the outline of the observer's pupil is missing, wherein, if the determination unit determines that the outline of the observer's pupil is missing, the detection unit detects the observer's gaze position by excluding a portion of the outline of the observer's pupil.

[0012] Another aspect of the present invention provides a control method for controlling a detection device, the control method comprising: capturing an image of an observer's eyeball; identifying a pupil center position based on the outline of the observer's pupil from the image of the eyeball acquired during the image capture; and using the pupil center position to detect the observer's gaze position; and determining the degree to which the outline of the pupil is obscured by the observer's eyelid, wherein, if the degree is determined to be a first degree, the observer's gaze position is detected using a pupil center position identified by approximating the outline of the pupil as a first circle; and wherein, if the degree is determined to be a second degree less than the first degree, the observer's gaze position is detected using a pupil center position identified by approximating the outline of the pupil as a second circle flatter than the first circle.

[0013] Another aspect of the present invention provides a control method for controlling a detection device, the control method comprising: capturing an image of an observer's eyeball; identifying a pupil center position based on the outline of the observer's pupil from the image of the eyeball acquired during the capture of the image; using the pupil center position to detect the observer's gaze position; and determining whether the outline of the observer's pupil is missing, wherein, if it is determined that the outline of the observer's pupil is missing, the observer's gaze position is detected by excluding a portion of the outline of the observer's pupil.

[0014] Other features of the invention will become apparent from the following description of embodiments with reference to the accompanying drawings. Attached Figure Description

[0015] Figure 1 This is an external schematic diagram of a camera device according to an embodiment of the present invention.

[0016] Figure 2 This is a block diagram of a camera device according to an embodiment of the present invention.

[0017] Figure 3This is a diagram illustrating the field of view in a viewfinder according to an embodiment of the present invention.

[0018] Figure 4A It is a schematic diagram of an eye image transmitted onto an eye image sensor, and Figure 4B This is a graph showing the brightness distribution of an eyeball image.

[0019] Figure 5 This is a diagram illustrating an example of the exclusion range of contour candidates according to a first embodiment of the present invention.

[0020] Figure 6 This is a flowchart of the calibration operation for gaze detection according to the first embodiment of the present invention.

[0021] Figure 7 This is a flowchart of the camera operation according to the first embodiment of the present invention.

[0022] Figure 8 This is a flowchart of the calibration operation for gaze detection according to the second embodiment of the present invention.

[0023] Figure 9 This is a diagram illustrating the determination of missing contour information according to a second embodiment of the present invention.

[0024] Figure 10 This is a flowchart of the calibration operation for gaze detection according to the third embodiment of the present invention.

[0025] Figure 11 This is a flowchart of the gaze detection operation according to the third embodiment of the present invention.

[0026] Figure 12 This is a flowchart of the calibration operation for gaze detection according to the fourth embodiment of the present invention.

[0027] Figure 13 This is a diagram illustrating the determination of missing contour information according to a fourth embodiment of the present invention.

[0028] Figure 14A This is a graph showing the error in the circle approximation when there are no regions with missing contour coordinates, and Figure 14B This is a graph showing the error in the circle approximation when there are regions with missing contour coordinates. Detailed Implementation

[0029] Embodiments of the present invention will now be described in detail based on the accompanying drawings. The various embodiments of the present invention described below can be implemented individually, or, where necessary or where a combination of elements or features from each embodiment is advantageous in a single embodiment, can be implemented as a combination of multiple embodiments or features thereof.

[0030] The first embodiment of the present invention will now be described. Figure 1 It is a cross-sectional view of the camera housing, and a schematic diagram showing the structure of a digital still camera 1 equipped with a line-of-sight detection unit according to a first embodiment of the present invention.

[0031] Figure 1 The image lens 1A of an interchangeable lens camera is shown. Although, for convenience, the interior of the image lens 1A is represented by the two lenses 101 and 102 in the first embodiment, it is known that the image lens 1A actually includes more lenses. Figure 1 The housing 1B of the camera body is shown. The structure of the units included in the housing 1B is as follows. An image sensor 2 is arranged on the image forming surface of the camera lens 1A of the digital still camera 1. The digital still camera 1 includes a central processing unit (CPU) 3 that controls the entire camera 1 and a memory unit 4 that records images captured by the image sensor 2. The digital still camera 1 also includes a display element 10 that includes a liquid crystal for displaying captured images, a display element driving circuit 11 that drives the display element 10, and an eyepiece lens 12 for observing the subject image displayed on the display element 10.

[0032] Light sources 13a and 13b illuminate the subject's eyeball 14 to detect the gaze direction based on the relationship between the reflected image obtained by corneal reflection light from a light source used in a single-lens reflex camera and the pupil. Light sources 13a and 13b are composed of infrared light-emitting diodes and are arranged near the eyepiece lens 12. The image of the illuminated eyeball 14 and the image obtained by corneal reflection light from light sources 13a and 13b pass through the eyepiece lens 12 and are reflected by a beam splitter 15. Then, a receiving lens 16 forms an image on an eye image sensor 17, in which columns of photoelectric elements such as complementary metal-oxide-semiconductor (CMOS) are arranged in a two-dimensional manner. The receiving lens 16 positions the subject's eyeball 14 and the eye image sensor 17 using a conjugate image formation relationship. Based on the positional relationship between the image of the eyeball 14 formed on the eye image sensor 17 and the image obtained by corneal reflection light from light sources 13a and 13b, the gaze direction is detected using a predetermined algorithm.

[0033] The camera lens 1A includes an aperture 111, an aperture drive device 112, a lens drive motor 113, and a lens drive member 114 consisting of drive gears. An optocoupler 115 detects the rotation of a pulse plate 116 linked to the lens drive member 114 and sends the detected rotation to a lens focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 by a predetermined amount based on information related to the detected rotation and information related to the amount of drive from the lens 1A of the camera 1, thereby moving the camera lens 1A to a focused position. A mounting contact 117 is an interface between a known camera and a known lens.

[0034] Alternatively, an image of the illuminated eyeball 14 can be directly captured by the eye image sensor 17 and the light-receiving lens 16 without passing through the eyepiece lens 12 and the beam splitter 15. In this case, the acquired image can be optically corrected at a position that does not interfere with the display element 10, by an amount corresponding to the arrangement of the eye image sensor 17 and the light-receiving lens 16.

[0035] Figure 2 This is a block diagram showing the electrical structure built into the digital still camera 1 having the above-described structure, and is related to... Figure 1 Identical components are designated by the same number. The line-of-sight detection circuit 201, metering circuit 202, autofocus detection circuit 203, signal input circuit 204, display element drive circuit 11, and illumination light source drive circuit 205 are connected to the CPU 3, which is a microcomputer built into the camera body. The CPU 3 is connected via... Figure 1 The mounting contact 117 shown communicates with the focus adjustment circuit 118 arranged in the camera lens 1A and the aperture control circuit 206 included in the aperture drive device 112. The memory unit 4 attached to the CPU 3 has a storage function for storing imaging signals from the image sensor 2 and the eye image sensor 17, and a storage function for storing gaze correction data used to correct individual differences in gaze.

[0036] The gaze detection circuit 201 performs analog-to-digital (A / D) conversion on the output obtained by forming an eye image from the eye image sensor 17, and sends information related to the image to the CPU 3. The CPU 3 extracts feature points of the eye image for gaze detection according to a predetermined algorithm, and further calculates the photographer's gaze from the position of the feature points.

[0037] Based on the signal obtained from the image sensor 2, which is also used as a light metering sensor, the light metering circuit 202 amplifies the output brightness signal corresponding to the brightness of the subject field, then performs logarithmic compression and A / D conversion on the output brightness signal, and sends the output brightness signal as subject field brightness information to the CPU 3.

[0038] The autofocus detection circuit 203 performs A / D conversion on the signal voltages from multiple pixels included in the CMOS of the image sensor 2 and used for phase difference detection, and sends the signal voltages to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to the focus detection point from the signals of the multiple pixels. This is a technique known as phase difference autofocus (AF).

[0039] Switch SW1 is connected to signal input circuit 204. The first stroke of the shutter release button (not shown) activates switch SW1, initiating metering, distance measurement, or line-of-sight detection operations on camera 1. Switch SW2 is also connected to signal input circuit 204. The second stroke of the shutter release button activates switch SW2, initiating the shutter release operation. The aforementioned signals are input to signal input circuit 204 and sent to CPU 3.

[0040] Optical system for line-of-sight detection

[0041] Figure 3 This diagram illustrates the principle of the gaze detection method and is equivalent to the method used for... Figure 1 A schematic diagram of the optical system for line-of-sight detection.

[0042] exist Figure 3 In this image, light sources 13a and 13b are light sources such as light-emitting diodes that emit invisible infrared light to the observer (the photographer). Light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of the receiving lens 16 and illuminate the observer's eyeball 14. A portion of the illuminated light reflected from the eyeball 14 is collected by the receiving lens 16.

[0043] <Principles of line-of-sight detection>

[0044] Next, refer to Figure 4A and Figure 4B To describe the principle of gaze detection operation.

[0045] Figure 4A This diagram illustrates an eye image obtained from the eye image sensor 17. From within the eye image, corneal reflection images Pd and Pe, representing light from light sources 13a and 13b, and the pupil center c, are detected. The rotation angle θ of the eyeball 14 can be calculated from these images. Next, based on the rotation angle θ and calibration data, the observer's gaze position on the display element 10 is obtained.

[0046] First, the detection methods for corneal reflection images Pd and Pe are described.

[0047] exist Figure 4AIn the diagram, the horizontal direction is the X-axis, and the vertical direction is the Y-axis. At this time, the coordinates of the positions where the corneal reflectance images Pd and Pe formed by the light from light sources 13a and 13b are (Xa,Ya) and (Xb,Yb), respectively.

[0048] Figure 4B It is shown that... Figure 4A A graph of brightness information associated with any single row (hereinafter referred to as the "search row") of an eye image.

[0049] exist Figure 4B In the brightness information, at the locations where corneal reflection images Pd and Pe from light sources 13a and 13b are formed, brightness levels greater than or equal to a brightness threshold TH2 are obtained at extremely high levels. The search row is changed, and the brightness level is judged sequentially to detect coordinates with brightness greater than or equal to the threshold TH2 throughout the entire eye image, and the centroid of these coordinates is calculated. This allows the detection of the coordinates (Xa, Ya) of corneal reflection image Pd and the coordinates (Xb, Yb) of corneal reflection image Pe.

[0050] <Detection method for determining the center position of the pupil>

[0051] Next, the method for detecting the pupil center c of pupil 141 will be described.

[0052] exist Figure 4B In the brightness information, the coordinates of the pupil contour are (Xc, Yc) and (Xd, Yd). Within the pupil 141 region, low-level brightness (less than or equal to the brightness threshold TH1) is obtained, except at coordinates (Xa, Ya) and (Xb, Yb). Conversely, in the region corresponding to the iris 143 outside the pupil 141, relatively high-level brightness (between brightness threshold TH1 and brightness threshold TH2) is obtained. That is, the portion with pixel coordinates (Xc, Yc) and (Xd, Yd) whose brightness changes at or below brightness threshold TH2 and is at the same brightness level as brightness threshold TH1 is identified as the pupil contour. The search row and search column are changed, and the brightness level is determined sequentially. This allows the detection of pupil contour coordinates across the entire eye image. Multiple pupil contour coordinate pairs are also referred to as "pupil contour coordinate sets." The detection of pupil contour coordinates is also called "contour detection."

[0053] By using the circular approximation method of the pupil's outline coordinates, the center coordinates c and radius r of pupil 141 were calculated.

[0054] If there are 20 pairs of first contour coordinates, and the first contour coordinates are represented by contour coordinates (Xi, Yi) (where i = 1 to 20), then the coordinates (X0, Y0) of the center c of the pupil 141 and the radius r are calculated by the following formula.

[0055]

[0056]

[0057]

[0058]

[0059] Next, the imaging magnification β of the eyeball is calculated. The magnification β is determined based on the position of the eyeball 14 relative to the light-receiving lens 16, and can be obtained essentially as a function of the distance between the corneal reflected image Pd and the corneal reflected image Pe.

[0060] The coordinates of the midpoint between the corneal reflective image Pd and the corneal reflective image Pe, as well as the coordinates of the center of curvature O of the cornea 142, almost match each other. Therefore, if the standard distance between the center of curvature O of the cornea 142 and the center c of the pupil 141 is Oc, the rotation angle θx of the optical axis of the eyeball 14 in the ZX plane can be obtained by the following formula.

[0061] β*Oc*SINθx≈{(Xa+Xb) / 2}-X0

[0062] Figure 3 This example illustrates the calculation of the rotation angle θx when the photographer's eyeball 14 rotates in a plane perpendicular to the Y-axis. The rotation angle θy when the photographer's eyeball 14 rotates in a plane perpendicular to the X-axis can be obtained using the following formula.

[0063] β*Oc*SINθy≈{(Ya+Yb) / 2}-Y0

[0064] Using the calculated rotation angles θx and θy, the position of the observer's gaze on the display element 10 is obtained. If the gaze point position has coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display element 10, the coordinates (Hx, Hy) can be calculated using the following formula.

[0065] Hx = m × (θx - θx_cal)

[0066] Hy = m × (θy - θy_cal)

[0067] At this point, the coefficient m is a constant determined by the structure of the camera's viewfinder optical system, and is a conversion coefficient that converts the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 141 on the display element 10. The coefficient m is predetermined and stored in the memory unit 4. θx_cal and θy_cal are the rotation angles calculated during the calibration operation and are stored in the memory unit 4 before the gaze detection routine begins.

[0068] The calibration operation is used to store an offset that reduces the positional deviation between the user's actual gaze location and the calculated estimated gaze point caused by individual differences in the shape of the human eyeball.

[0069] During calibration, a calibration image is displayed on display element 10. The image is the target frame that the photographer is looking at and is displayed in the center of display element 10. The rotation angles θx_cal and θy_cal at this time are acquired and stored as offsets in memory unit 4.

[0070] The accuracy of calibration can be improved by displaying the target box at multiple locations and interpolating multiple locations using rotation angles at multiple locations.

[0071] This is a description of the principle behind eye-tracking detection.

[0072] refer to Figures 5 to 7 The gaze detection operation according to the first embodiment will be described.

[0073] <Line of sight detection during calibration>

[0074] Figure 6 This is a flowchart illustrating the calibration operation for gaze detection. The operations shown in the flowchart are executed under the control of CPU 3.

[0075] In step S101, the calibration image is displayed on the display element 10.

[0076] In step S102, light sources 13a and 13b emit infrared light to the observer's eyeball 14. The eyeball image irradiated by the infrared light is formed on the eyeball image sensor 17 through the light-receiving lens 16 and is photoelectrically converted by the eyeball image sensor 17. The eyeball image can be processed as an electrical signal.

[0077] In step S103, CPU 3 receives the eye image signal obtained by eye image sensor 17.

[0078] In step S104, based on information related to the eye image signal obtained through the operation in step S103, the following is obtained: Figure 3The coordinates of the corneal reflection images Pd and Pe from light sources 13a and 13b, as well as the first contour coordinates, are shown.

[0079] In step S105, the center coordinates c and radius r of the pupil 141 are calculated using the outline coordinates of the pupil 141 obtained through the operation in step S104 as the first outline coordinates by the circular approximation method.

[0080] In step S106, the second contour coordinates are selected from the information related to the "first contour coordinates" and the "center coordinates c and radius r calculated from the first contour coordinates".

[0081] Above the pupil 141, contour information may be missing due to the eyelid. Therefore, in the obtained contour coordinates, the contour coordinates of the upper part are removed in advance, thereby reducing the error in the center coordinates based on the presence or absence of missing information.

[0082] Figure 5 This is a diagram showing an example of the selection range for contour coordinates.

[0083] The coordinates included in the shaded area at a distance r / 2 or more from the center coordinate in the Y-axis direction are excluded, and the coordinates excluding those in the shaded area are selected as the second contour coordinates. In this example, the exclusion range is limited to the area above r / 2, but this can be changed depending on the size or image formation position.

[0084] In step S107, the center coordinates c' and radius r' of the pupil 141 are calculated from the second contour coordinates selected in step S106 using the circular approximation method.

[0085] In step S108, the imaging magnification β of the eye image is calculated.

[0086] In step S109, the rotation angles θx and θy of the optical axis of the eyeball 14 in the ZX plane are calculated.

[0087] In step S110, the rotation angles θx and θy calculated through the operation in step S109 are stored in memory unit 4 as corrected values ​​θx_cal and θy_cal.

[0088] <Camera Operation>

[0089] exist Figure 6 After the calibration operation for the gaze detection described in the document, the camera operation is performed.

[0090] Figure 7 This is a flowchart illustrating the image capture operation of camera 1 with gaze detection function. The operations shown in the flowchart are executed under the control of CPU 3.

[0091] If camera 1 is powered on and the process begins, in step S201, image sensor 2 is driven and image acquisition begins. CPU 3 displays the acquired image on display element 10.

[0092] In step S202, it is determined whether the camera 1 has been powered off. If the camera has been powered off ("Yes" in step S202), the process ends. If the camera 1 has not been powered off ("No" in step S202), the process proceeds to step S203.

[0093] In step S203, a line-of-sight detection operation is performed. In step S203, a line-of-sight detection operation is performed relative to a reference line. Figure 6 The operation is similar to steps S102 to S109 in the calibration operation described in the flowchart. Additionally, in the gaze detection operation during image capture, the rotation angles θx and θy of the eyeball 14 are calculated from the pupil center, which is calculated by excluding the upper part of the pupil contour obtained from the eyeball image sensor 17.

[0094] In step S204, the rotation angles θx and θy of the eyeball 14 detected by the operation in step S203 are corrected. The rotation angles θx and θy and the corrected values ​​θx_cal and θy_cal stored in the memory unit 4 in step S110 are read, and the coordinate values ​​Hx and Hy are calculated and set as the fixation point position.

[0095] In step S205, an AF frame is displayed at the gaze point position on the display element 10.

[0096] In step S206, it is determined whether the shutter release button was pressed by the photographer to turn on switch SW1. If switch SW1 is turned on (yes in step S206), the process proceeds to step S207. If switch SW1 is not turned on (no in step S206), the process returns to step S201. In step S201, the image display and gaze detection operations are repeated.

[0097] In step S207, the AF operation is performed.

[0098] Image sensor 2 includes multiple pixels for phase difference detection. Using the known technique of image plane phase difference AF, CPU 3 calculates the focus state of the subject corresponding to the AF frame determined in step S205 from the signals of the multiple pixels, and controls and adjusts the positions of lens 101 and lens 102.

[0099] In step S208, it is determined whether the shutter release button should be pressed further and switch SW2 should be turned on. If switch SW2 is turned on (yes in step S208), the process proceeds to step S209. If switch SW2 is not turned on (no in step S208), the process returns to step S206. In step S206, the AF operation is performed again at the same location.

[0100] In step S209, the image sensor 2 is driven to acquire an image and record the image on a recording medium (not shown). Then, the process returns to the operation in step S201.

[0101] As described above, through Figure 6 The operations in steps S106 and S107 exclude the coordinates in the upper part of the contour that may be missing due to the eyelid, and then perform a circle approximation calculation (steps S106 and S107).

[0102] If the pupil is elliptical, excluding the upper part of the outline causes a shift in the center coordinates. Therefore, the same exclusion process is performed during both calibration and imaging to counteract the effects of the exclusion.

[0103] In the first embodiment, a method is described for assuming a range of missing contour coordinates under the influence of the eyelid and always excluding contour information related to that range. In the second embodiment, a method for dynamically switching the missing range is described.

[0104] The camera structure and line-of-sight detection device according to the second embodiment of the present invention are the same as those in the first embodiment. Figures 1 to 4A and Figure 4B The structure of the camera is similar to that of the line-of-sight detection device, therefore, it will not be described here.

[0105] <Line of sight detection during calibration>

[0106] Figure 8 This is a flowchart illustrating the calibration operation for gaze detection. The operations shown in the flowchart are executed under the control of CPU 3.

[0107] The operation of steps S301 to S305 and Figure 6 The operations of steps S101 to S105 are similar, so they will not be described.

[0108] In step S306, the missing range is detected from the "first contour coordinates" and the "center coordinates c and radius r calculated from the first contour coordinates", and the second contour coordinates are selected.

[0109] Figure 9 This is a diagram illustrating an example of the selection range of contour coordinates according to the second embodiment.

[0110] The circle formed by the "first contour coordinates" and the "center coordinates c and radius r calculated from the first contour coordinates" is divided into eight sector regions (1) to (8). For each of the sectors (1) to (8), the number of contour coordinate pairs is counted. Sectors that are judged to have fewer than or equal to a predetermined number of contour coordinate pairs are excluded from the contour candidates. In this case, sectors (1) and (8) are excluded, and the coordinates present in sectors (2) to (7) are selected as the second contour coordinates.

[0111] In step S307, the center coordinates c' and radius r' of the pupil 141 are calculated from the second contour coordinates selected in step S306 using the circular approximation method.

[0112] In step S308, the imaging magnification β of the eye image is calculated.

[0113] In step S309, the rotation angles θx and θy of the optical axis of the eyeball 14 are obtained.

[0114] In step S310, the rotation angles θx and θy calculated through the operation in step S309 are stored in memory unit 4 as corrected values ​​θx_cal and θy_cal. Information related to the region determined in step S306 is also stored in memory unit 4.

[0115] <Camera Operation>

[0116] exist Figure 8 After the calibration operation for gaze detection described above, the camera operation is performed. This camera operation is related to... Figure 7 The camera operation is similar to that in the flowchart, so it will not be described.

[0117] As described above, during calibration, regions with missing contour coordinates are detected, these coordinates are excluded, and then a perfect circle approximation is performed.

[0118] If the pupil is elliptical, the exclusion of the missing area during calibration will cause a shift in the center coordinates. Therefore, the same area is excluded during both calibration and imaging to offset the effect of the exclusion.

[0119] If the pupil outline cannot be detected normally not only due to the influence of the eyelids but also due to the influence of external light, then not only the upper part of the outline may be missing, but also the left or right part of the outline may be missing. Even under these circumstances, gaze detection can still be performed well.

[0120] In the third embodiment, the learned pupil shape is used for elliptical approximation, thereby reducing the impact of eyelid-induced defects on the accuracy of visual line detection.

[0121] It is known that pupil shape varies from person to person, and there are people whose pupil shape is not a perfect circle but an ellipse.

[0122] Figure 14A This is a graph showing the center coordinates of an elliptical pupil shape detected by approximation using a perfect circle, based on information related to the pupil's edge. Dashed lines and "·" markers indicate the actual pupil outline and its center coordinates, while "×" markers indicate the coordinates of the pupil outline determined based on the output signal from the area sensor. Solid lines and "+" markers indicate the approximate circle calculated using a perfect circle approximation based on the pupil outline coordinates and the center coordinates of the approximate circle. Figure 14A In the diagram, contour information exists above, below, to the left, and to the right of the pupil's center. Therefore, although the pupil's shape is actually elliptical, the "·" and "+" symbols almost match each other when approximated as a circle using the least squares method.

[0123] on the other hand, Figure 14B It shows that it detected and Figure 14A This describes a pupil with a similar elliptical shape, but with the outline partially missing and only the lower half of the outline present. In fact, due to the influence of the eyelids, the upper part of the pupil is often missing.

[0124] As mentioned above, if the coordinates of the outline edge of the lower half of the elongated ellipse are only calculated using a circular approximation, the center of gravity shifts downwards from the actual pupil center.

[0125] Therefore, in this embodiment, as described above, the learned pupil shape is used for elliptical approximation, thereby reducing the impact of eyelid-induced defects on the detection accuracy of the line of sight.

[0126] The structure of the camera and line-of-sight detection device according to the third embodiment is the same as that in the first embodiment. Figures 1 to 4A and Figure 4B The camera and gaze detection device in this example have similar structures, therefore, they will not be described here. <Identification of the pupil center position using an elliptical approximation>

[0127] Next, we will describe the elliptical approximation.

[0128] The general formula for an ellipse is as follows.

[0129]

[0130] X0, Y0: Coordinates of the center of the ellipse

[0131] θ: Inclination of the ellipse

[0132] a: Length of the ellipse along the X-axis

[0133] b: Length of the ellipse along the Y-axis

[0134] The expansion of the above formula and the substitution of unknowns can be expressed as follows.

[0135] ∑(Xi 2 +A*Yi 2 +B*Xi+C*Yi+D) 2 =0 Formula (1)

[0136] The partial differentials from A to E can be represented by the following matrix.

[0137]

[0138] Based on points A to E, the ellipse parameters can be obtained using the following formula.

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] If the photographer is not looking at the digital still camera 1 with their eyeball 14 tilted relative to the digital still camera 1 (i.e., if θ = 0), then the above formula is transformed as follows.

[0145]

[0146] Based on points A to D, the ellipse parameters are obtained using the following formula.

[0147]

[0148]

[0149]

[0150]

[0151]

[0152] It should be understood that A is the equivalent value of the flattening (a / b) of an ellipse.

[0153] Furthermore, in order to make it possible to obtain the flattening u = a / b of the stored ellipse by adding the standardization term to the least squares in formula (1), formula (1) is transformed as follows.

[0154] ∑(Xi 2 +A*Yi 2 +B*Xi+C*Yi+D) 2 +λ*(Au 2 ) 2 =0

[0155] u: The flatness (a / b) stored

[0156] λ: Influence of the standardized term

[0157] A: Flattening of the ellipse (a / b) 2

[0158] If we transform the above formula using partial differential equations...

[0159]

[0160] The formulas above using the least squares method are used to calculate A to D, and based on the calculation results, the ellipse parameters are obtained using the following formula.

[0161]

[0162]

[0163]

[0164]

[0165]

[0166] Using the above calculation formula, the error in determining the center coordinates based on the absence of contour information caused by the eyelid can be reduced.

[0167] Next, a flowchart using the above calculation formula is described.

[0168] <Line of sight detection during calibration>

[0169] Figure 10 This is a flowchart illustrating the calibration operation for gaze detection, which includes learning an elliptical shape. The operations shown in the flowchart are executed under the control of CPU 3.

[0170] The operation of steps S401 to S405 and Figure 8 The operations of steps S301 to S305 are similar, so they will not be described.

[0171] In step S406, with Figure 8 Similar to step S306, the missing range is detected from the "first contour coordinates" and the "center coordinates c and radius r calculated from the first contour coordinates".

[0172] Similar to the second embodiment, for Figure 9 For each of regions (1) to (8), the number of contour coordinate pairs is counted. If there is a region where the number of contour coordinate pairs is less than or equal to a predetermined number, it is determined that there is a missing range.

[0173] In step S407, if it is determined that there is no missing range ("No" in step S407), then in step S408, the center c' and flattening u of the ellipse are calculated using formulas (2-1) to (2-6). The flattening u is stored in memory unit 4.

[0174] In step S409, similar to step S308, the imaging magnification β of the eye image is calculated.

[0175] In step S410, similar to step S309, the rotation angles θx and θy of the optical axis of the eyeball 14 are obtained. If no missing range exists, the center c' of the pupil 141, calculated by approximation using an ellipse in step S407, is used. If a missing range exists, the center c of the pupil 141, calculated by approximation using a perfect circle in step S405, is used.

[0176] In step S411, similar to step S310, the calculated rotation angles θx and θy are stored in memory unit 4 as corrected values ​​θx_cal and θy_cal.

[0177] <Camera Operation>

[0178] exist Figure 10 After the calibration operation for the gaze detection described in the document, the camera operation is performed.

[0179] In addition to the processing in step S203, the camera operation is also related to... Figure 7 The camera operation is similar to that in the flowchart, so it will not be described.

[0180] <Line of sight detection>

[0181] Figure 11 yes Figure 7 The flowchart for the gaze detection operation in step S203 is shown. The operation described in the flowchart is executed under the control of CPU 3.

[0182] The operation of steps S501 to S504 and Figure 8 The operations of steps S301 to S304 are similar, so they will not be described.

[0183] In step S505, the center coordinates c' and flattening u' of the pupil 141 are calculated using the elliptic approximation method of formulas (3-1) to (3-6), using the flattening u held in memory unit 4 and the contour coordinates of the pupil 141 obtained as the first contour coordinates through the operation of step S504.

[0184] In step S506, with Figure 8 Similarly, in step S306, the missing range is detected from the "first contour coordinates" and the "center coordinates c' of the ellipse calculated from the first contour coordinates and the length in the X-axis direction / the length in the Y-axis direction".

[0185] for Figure 9 For each of regions (1) to (8), the number of contour coordinate pairs is counted. If there is a region where the number of contour coordinate pairs is less than or equal to a predetermined number, it is determined that there is a missing range.

[0186] In step S507, if it is determined that there is no missing range (No in step S507), then in step S508, the weighted average of the flatness u stored in memory unit 4 and the flatness u' calculated in step S505 is calculated, and the flatness u stored in memory unit 4 is updated.

[0187] The flattening of the pupil 141 changes between when the photographer is looking straight ahead and when the photographer looks up, left, and right by rotating the pupil 141. Therefore, the position of the pupil relative to the edge of the eyelid can be detected, and the flattening u at each position can be stored in the memory unit 4.

[0188] Furthermore, the diameter of the pupil 141 varies depending on the brightness. Therefore, the flatness u can be stored in the memory unit 4 relative to the size of each pupil diameter.

[0189] In step S509, similar to step S308, the imaging magnification β of the eye image is calculated.

[0190] In step S510, similar to step S309, the rotation angles θx and θy of the optical axis of the eyeball 14 are obtained. In this case, the center c' of the pupil 141, which was calculated by ellipse approximation in step S505, is used.

[0191] As described above, by repeatedly learning the shape of the pupil 141 and performing an elliptical approximation using the learned results, the effects of missing parts can be reduced.

[0192] In the fourth embodiment, the circular approximation and the elliptical approximation are switched according to the degree of missing pupil contour, thereby reducing the impact of the missing pupil caused by the eyelid on the detection accuracy of the line of sight.

[0193] As mentioned above, it is known that pupil shape varies from person to person, and there are individuals whose pupil shape is not a perfect circle but an ellipse. In any pupil shape, if there are no missing parts, the center coordinates of pupil 141 can be obtained by performing an ellipse approximation using the least squares method.

[0194] However, if the outline is like Figure 14B If the area shown is missing, the ellipse approximation has too much freedom, resulting in an ellipse that can be either longitudinally or laterally elongated. Consequently, the center coordinates may be less accurate compared to the circle approximation.

[0195] Therefore, it is necessary to determine the presence or absence of defects in the contour and appropriately switch between circular and elliptical approximations to reduce errors in line-of-sight detection.

[0196] The structure of the camera and line-of-sight detection device according to the fourth embodiment is the same as that in the first embodiment. Figures 1 to 4A and Figure 4B The camera and gaze detection device in the image have similar structures, so they will not be described here.

[0197] Next, a flowchart using the above calculation formula will be described.

[0198] <Line of sight detection during calibration>

[0199] Figure 12 This is a flowchart illustrating the calibration operation for gaze detection.

[0200] The flowchart operations are executed under the control of CPU 3.

[0201] The operation of steps S601 to S605 and Figure 8 The operations of steps S301 to S305 are similar, therefore they will not be described.

[0202] In step S606, the center c' of the ellipse is calculated using formulas (2-1) to (2-5).

[0203] In step S607, the presence or absence of missing values ​​is detected from the "first contour coordinates" and the "center coordinates c' and radius r calculated from the first contour coordinates".

[0204] Figure 13 This is a diagram showing an example of the detection of the presence or absence of missing items.

[0205] The number of first contour coordinate pairs is counted in the portion of the center coordinate above r / 5 in the upward direction of the Y-axis. If the counted number is greater than or equal to a predetermined number, it is determined that there are no missing pairs in the upward direction of the Y-axis. Similarly, it is determined whether there are missing pairs in the downward direction of the Y-axis and the left and right directions of the X-axis. If there are no missing pairs in any or all of the four directions, it is determined that there are no missing pairs. That is, thresholds are set for any or all of the upward, downward, left, and right directions, and the missing contours are determined.

[0206] The size of pupil 141 changes according to light intensity. Therefore, the threshold for the number of counts can be changed according to the size r of pupil 141.

[0207] In step S608, similar to step S308, the imaging magnification β of the eye image is calculated.

[0208] In step S609, similar to step S309, the rotation angles θx and θy of the optical axis of the eyeball 14 are obtained. If no missing range exists, the center c' of the pupil 141, calculated by approximation using an ellipse in step S606, is used. If a missing range exists, the center c of the pupil 141, calculated by approximation using a perfect circle in step S605, is used.

[0209] <Description of camera operation>

[0210] exist Figure 12 After the calibration operation for the gaze detection described in the document, the camera operation is performed.

[0211] Apart from Figure 7 In addition to the processing in step S203, the camera operation and Figure 7 The camera operation is similar to that in the flowchart, so it will not be described.

[0212] In step S203, a reference is performed. Figure 12 The operation is similar to steps S602 to S609 in the calibration operation described in the flowchart. Additionally, in the gaze detection operation during image capture, the approximation of a perfect circle and an ellipse is switched based on the degree of missing pupil contour.

[0213] As described above, the presence or absence of missing data is detected, and the circular approximation and elliptical approximation are switched based on the detection results, thereby reducing the impact of missing data.

[0214] If the circle approximation and ellipse approximation are switched frequently, the results will vary. Therefore, a hysteresis can be set to make it less likely that the circle approximation and ellipse approximation will be switched. For example, a threshold r / 5 and a threshold for the counted quantity can be set based on previous results of the missing / absent judgment. Alternatively, if the same result of missing / absent judgment continues for a predetermined number or more, the approximation method can be switched.

[0215] Furthermore, if the calibration process determines that an elliptical pupil should be approximated as a perfect circle, and if this elliptical approximation is applied during the imaging operation, the center coordinates will shift. Therefore, if there is a defect in the calibration process, it can be determined that a perfect circle approximation should also be applied during the imaging operation.

[0216] <Other Embodiments>

[0217] CPU 3 can control display element 10 to display an indicator that indicates the gaze position corresponding to the gaze detected as a result of gaze detection in the above embodiment.

[0218] In the operations described in the flowcharts of the above embodiments, the order of the steps to be performed can be appropriately changed to achieve a similar purpose.

[0219] As long as the device has a gaze detection function, the device using this embodiment is not limited to the camera described in this embodiment. For example, this embodiment can be applied to head-mounted displays.

[0220] Although embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and modifications and changes can be made in various ways within the scope of the present invention.

[0221] Other embodiments

[0222] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0223] Although the invention has been described with reference to embodiments, it should be understood that the invention is not limited to the disclosed embodiments. The scope of the appended claims is to be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. A detection device, comprising: A camera unit configured to capture images of the observer's eyeballs; A detection unit is configured to identify the pupil center position based on the outline of the observer's pupil according to an image of the eyeball acquired from the camera unit, and to use the pupil center position to detect the observer's gaze position; as well as A determination unit is configured to determine the degree to which the outline of the pupil is obscured by the observer's eyelids. The determination unit determines the degree by counting the number of points indicating the outline of the pupil within a region defined based on the pupil's center position and radius, and comparing the counted number with a threshold based on the radius. The pupil's center position and radius are identified by approximating the points indicating the pupil's outline as circles. Wherein, when the judgment unit determines that the degree is the first degree, the detection unit uses the pupil center position identified by approximating the pupil's outline as a first circle to detect the observer's gaze position, where the first circle is a perfect circle. In cases where the judgment unit determines that the degree is a second degree less than the first degree, the detection unit uses the pupil center position identified by approximating the pupil outline as an ellipse flatter than the first circle to detect the observer's gaze position.

2. The detection device according to claim 1, wherein, The judgment unit determines the degree to which the outline of the pupil is obscured by the observer's eyelids based on the pupil center position identified by approximating the points indicating the outline of the pupil as the first circle, and by comparing the number of points indicating the outline of the pupil with a threshold.

3. The detection device according to claim 2, wherein, The threshold is based on the radius of a second circle, which is obtained by approximating the first circle with points indicating the outline of the pupil.

4. The detection device according to claim 1, wherein, The information indicating the flattening of the ellipse is stored, and the information is used to identify the center position of the pupil.

5. The detection device according to claim 4, wherein, The information indicating the flattening of the ellipse is stored based on the position of the eyelid and the center position of the pupil.

6. A control method for controlling a detection device, the control method comprising: To capture an image of the observer's eyeball; The pupil center position is identified based on the outline of the observer's pupil using an image of the eyeball acquired during the image capture, and the pupil center position is used to detect the observer's gaze position. as well as Determine the degree to which the outline of the pupil is obscured by the observer's eyelids. In this determination, the degree is assessed by counting the number of points indicating the outline of the pupil within a region defined based on the pupil's center position and radius, and the counted number is compared to a threshold based on the radius. The pupil's center position and radius are identified by approximating the points indicating the pupil's outline as circles. Wherein, if the degree is determined to be the first degree, the observer's gaze position is detected using the pupil center position identified by approximating the pupil's outline as a first circle, where the first circle is a perfect circle. In cases where the degree is determined to be a second degree less than the first degree, the observer's gaze position is detected by using the pupil center position identified by approximating the pupil's outline as an ellipse flatter than the first circle.

7. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of the method according to claim 6.

8. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to claim 6.

Images