Electronic device and control method thereof

Abstract

An electronic device and a control method thereof are provided. The electronic device includes a detection unit configured to perform gaze detection based on an eye image in which an eye looking at a display unit is photographed, and a control unit configured to perform control to give a predetermined notification about a method of adjusting a viewing state in which the display unit is visually perceived based on at least one of a position of a pupil image in the eye image and a number of corneal reflection images in the eye image.

Description

Technical Field

[0001] This invention relates to an electronic device with a gaze detection function. Background Technology

[0002] Cameras (including video cameras) that can detect a user's gaze (gaze direction) using gaze detection and can select the focus point based on the gaze detection results have been put into use.

[0003] Japanese Patent Application Publication No. 2018-32198 discloses a technology for displaying an image of a user's eyes on a display device. Users can adjust the position of their face, the orientation of their face, the position of their glasses, and the orientation of their glasses based on the eye image, in order to limit the reduction in gaze detection accuracy that may be caused by unnecessary light, such as light reflected from the glasses.

[0004] However, using the conventional technology disclosed in Japanese Patent Application Publication No. 2018-32198, users unfamiliar with gaze detection functions may not be able to easily determine the specific adjustment methods to prevent a decrease in gaze detection accuracy, and therefore may not be able to easily make adjustments. As a result, the eye may be located outside the gaze detection area (the area capable of gaze detection), or unnecessary light, such as light reflected from the glasses, may not be eliminated. This reduces gaze detection accuracy. Summary of the Invention

[0005] The present invention provides an electronic device that allows users to easily adjust the position of the face, the orientation of the face, the position of the glasses, the orientation of the glasses, etc., thereby enabling gaze detection with high accuracy.

[0006] An electronic device is capable of acquiring an eye image of an eye looking at a display unit. The electronic device includes: a detection unit configured to perform gaze detection based on the eye image; and a control unit configured to control the device to provide a predetermined notification related to a method of adjusting the viewing state based on at least one of the position of the pupil image in the eye image and the number of corneal reflection images in the eye image, wherein the display unit is visually perceived in the viewing state.

[0007] Other features of the invention will become apparent from the description of the following exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0008] Figure 1A and Figure 1B This is an external view of the camera according to this embodiment;

[0009] Figure 2 This is a block diagram of a camera according to this embodiment;

[0010] Figure 3This is a cross-sectional view of the camera according to this embodiment;

[0011] Figure 4A and Figure 4B This is a diagram showing the EVF portion of the camera according to this embodiment;

[0012] Figure 5 This is a diagram showing the optical path of the light emitted by the infrared LED according to this embodiment;

[0013] Figure 6 This is a diagram used to illustrate the principle of the line-of-sight detection method according to this embodiment;

[0014] Figure 7A and Figure 7B This is a diagram showing an eye image according to this embodiment;

[0015] Figure 8 This is a flowchart of the line-of-sight detection operation according to this embodiment;

[0016] Figure 9 This is a flowchart of the calibration operation according to this embodiment;

[0017] Figure 10A , Figure 10B , Figure 10E , Figure 10G , Figure 10I and Figure 10J This is a diagram showing a display image according to this embodiment;

[0018] Figure 10C , Figure 10D , Figure 10F and Figure 10H This is a diagram showing an eye image according to this embodiment;

[0019] Figure 11 It is a cross-sectional view of a camera based on a variant example; and

[0020] Figure 12A and Figure 12B This is a diagram showing an eye image according to a variant example. Detailed Implementation

[0021] [Implementation Method]

[0022] Preferred embodiments of the present invention are described below with reference to the accompanying drawings.

[0023] <Configuration Description>

[0024] Figure 1A and Figure 1BThe appearance of a camera 1 (digital still camera; camera with interchangeable lenses) according to this embodiment is shown. The invention is also applicable to devices that display information such as images and text, and to any electronic device capable of detecting the gaze of a user visually perceiving an optical image through an eyepiece optical system. Examples of such electronic devices include mobile phones, game consoles, tablet terminals, personal computers, information terminals in the form of watches or glasses, head-mounted displays, and binoculars. The invention is applicable to any electronic device capable of obtaining an image of the eye looking at the display.

[0025] Figure 1A It is a frontal 3D view. Figure 1B It's a 3D view of the back. For example... Figure 1A As shown, camera 1 has a camera lens unit 1A and a camera housing 1B. Camera housing 1B includes a release button 34, which is an operating component that receives recording operations from the user (photographer). Figure 1B As shown, the rear side of the camera housing 1B includes an eyepiece window frame 121 through which the user observes the display panel 6 located inside the camera housing 1B. The display panel 6 will be described later. The eyepiece window frame 121 forms a viewing opening 12 and protrudes outward (towards the rear) relative to the camera housing 1B. The rear side of the camera housing 1B also includes operating members 41 to 43, which receive various operations from the user. For example, operating member 41 is a touch panel that receives touch operations, operating member 42 is a joystick that can be pressed to tilt in different directions, and operating member 43 is a four-way key that can be pressed in four directions. Operating member 41 (touch panel) includes a display panel such as a liquid crystal panel and has the function of displaying images on the display panel.

[0026] Figure 2 This is a block diagram showing the configuration of camera 1.

[0027] The imaging element 2 can be an imaging device such as a CCD or CMOS sensor. The optical system of the imaging lens unit 1A forms an optical image on the image plane of the imaging element 2. The imaging element 2 performs photoelectric conversion on the optical image and outputs the obtained analog image signal to an A / D conversion unit (not shown). The A / D conversion unit performs analog-to-digital conversion on the analog image signal obtained by the imaging element 2 and outputs the converted signal as image data.

[0028] The camera lens unit 1A comprises an optical system including a zoom lens, a focusing lens, and an aperture. When mounted on the camera housing 1B, the camera lens unit 1A guides light from the subject to the image sensor 2, forming an image of the subject on the image plane of the image sensor 2. The aperture control unit 118, the focus adjustment unit 119, and the zoom control unit 120 each receive command signals from the CPU 3 via mounting contacts 117, and drive and control the aperture, focusing lens, and zoom lens according to the command signals.

[0029] The CPU 3 in the camera housing 1B reads the control program for the block of the camera housing 1B from the ROM of the memory unit 4, loads the control program into the RAM of the memory unit 4, and executes the control program. Thus, the CPU 3 controls the operation of the block of the camera housing 1B. The CPU 3 is connected to, for example, a gaze detection unit 201, a metering unit 202, an autofocus detection unit 203, a signal input unit 204, a display device driver unit 210, and a light source driver unit 205. The CPU 3 transmits signals via mounting contacts 117 to the aperture control unit 118, the focus adjustment unit 119, and the zoom control unit 120 arranged in the camera lens unit 1A. In this embodiment, the memory unit 4 has the function of storing the imaging signals received from the imaging element 2 and the gaze detection sensor 30.

[0030] When an image of the eyeball is formed on the gaze detection sensor 30, the gaze detection unit 201 performs analog-to-digital conversion on the output (eye image) generated by the gaze detection sensor 30. The conversion result is sent to the CPU 3. The CPU 3 extracts the feature points required for gaze detection from the eye image according to a predetermined algorithm, which will be described later, and calculates the user's gaze (gaze point in the image for visual perception) from the location of the feature points.

[0031] The metering unit 202 processes the signal obtained from the imaging element 2, which also functions as a metering sensor, using methods such as amplification, logarithmic compression, and A / D conversion. This signal is a brightness signal corresponding to the brightness of the subject's field of view. The processing result is sent to the CPU 3 as field-of-view brightness information.

[0032] The automatic focus detection unit 203 performs analog-to-digital conversion on the signal voltages received from multiple detection elements (multiple pixels) disposed in the imaging element 2 (such as a CCD) for detecting phase differences. The converted voltages are sent to the CPU 3. Based on the signals received from the detection elements, the CPU 3 calculates the distance to the subject corresponding to the focus detection point. This is a conventional technique known as image-plane phase detection AF. For example, assuming in this embodiment, the viewfinder image (image for visual perception) is segmented, and a focus detection point exists at each of 180 different locations on the image plane.

[0033] The light source driving unit 205 drives the infrared LEDs 18, 19, 22-27, which will be described later, based on signals (instructions) from the CPU 3.

[0034] The image processing unit 206 performs various image processing operations on the image data stored in RAM. The image processing unit 206 performs various image processing operations for developing, displaying, and recording digital image data, such as correction of pixel defects caused by optical systems or camera elements, demosaic, white balance correction, color interpolation, and gamma processing.

[0035] Signal input unit 204 is connected to switches SW1 and SW2. Switch SW1 is used to initiate operations of camera 1 such as metering, rangefinding, and line-of-sight detection, and is turned on by releasing the first stroke of button 34. Switch SW2 is used to initiate video recording operations and is turned on by releasing the second stroke of button 34. ON signals from switches SW1 and SW2 are input to signal input unit 204 and transmitted to CPU 3. Signal input unit 204 also receives signals from... Figure 1B The operation inputs of the operation components 41 (touch panel), 42 (joystick), and 43 (four-way key) are shown.

[0036] The recording / output unit 207 records data, including image data, on a recording medium such as a removable memory card, and outputs the data to an external device via an external interface.

[0037] The display device driving unit 210 drives the display device 209 based on signals from the CPU 3. The display device 209 includes display panels 5 and 6, which will be described later.

[0038] Figure 3 It is camera 1 along the... Figure 1A The figure shows a cross-sectional view of camera 1 taken by the YZ plane formed by the Y-axis and Z-axis. This figure conceptually illustrates the configuration of camera 1.

[0039] The shutter 32 and the imaging element 2 are arranged side by side along the optical axis of the camera lens unit 1A.

[0040] The rear side of the camera housing 1B includes a display panel 5, which displays menus and images for operating the camera 1 and viewing and editing images acquired by the camera 1. The display panel 5 may be, for example, a backlit liquid crystal panel or an organic EL panel.

[0041] The EVF housed in the camera housing 1B has the functions of a conventional EVF, displaying menus and images like the display panel 5. Furthermore, the EVF can detect the gaze of the user observing it, allowing the control of the camera 1 to reflect the detection results.

[0042] When the user observes the viewfinder, display panel 6 displays information similar to that of display panel 5 (menu display and image display for operating camera 1 and viewing / editing images acquired by camera 1). Display panel 6 can be, for example, a backlit LCD panel or an OLED panel. Similar to images captured by a conventional camera, display panel 6 is a rectangle whose X-axis (horizontal direction) is longer than its Y-axis (vertical direction) in an aspect ratio of, for example, 3:2, 4:3, or 16:9.

[0043] Panel retainer 7 holds display panel 6. Display panel 6 and panel retainer 7 are bonded together to form display panel unit 8.

[0044] The first optical path splitting prism 9 and the second optical path splitting prism 10 are attached and bonded to each other to form an optical path splitting prism unit 11 (optical path splitting component). The optical path splitting prism unit 11 guides light from the display panel 6 to the eyepiece window 17 disposed in the viewing port 12, and guides light from the eyepiece window 17 in the opposite direction to the gaze detection sensor 30. The light from the eyepiece window 17 includes light reflected from the eye (pupil).

[0045] The display panel unit 8 and the optical path splitting prism unit 11 are fixed together and formed as one unit through the mask 33.

[0046] The eyepiece optical system 16 includes lens 13 (G1), lens 14 (G2), and lens 15 (G3).

[0047] The eyepiece window 17 is a transparent component that allows visible light to pass through. The image displayed on the display panel unit 8 is observed through the optical path splitting prism unit 11, the eyepiece optical system 16, and the eyepiece window 17.

[0048] Illumination windows 20 and 21 are used to conceal infrared LEDs 18, 19, and 22-27, making them invisible from the outside. Illumination windows 20 and 21 are made of resin that absorbs visible light but transmits infrared light.

[0049] Figure 4A This is a perspective view showing the configuration of the EVF portion of camera 1. Figure 4B This is a cross-sectional view of the optical axis of the EVF section.

[0050] Infrared LEDs 18, 19, and 22–27 are light sources that emit infrared light. Infrared LEDs 18, 19, 23, and 25 are infrared LEDs used for short-range lighting. Infrared LEDs 22, 24, 26, and 27 are infrared LEDs used for long-range lighting. Light sources other than infrared LEDs can also be used.

[0051] The gaze detection optical system, including aperture 28 and gaze imaging lens 29, guides infrared reflected light from eyepiece window 17 to gaze detection sensor 30 via optical path splitting prism unit 11. Gaze detection sensor 30 is a solid-state image sensor such as CCD or CMOS.

[0052] For example, light from at least one of the infrared LEDs 18, 19, and 22-27 illuminates the eye of a user viewing the viewfinder. In this case, as... Figure 4B As shown in optical path 31a, the optical image (eye image) of the illuminated eyeball passes through eyepiece window 17, G3 lens 15, G2 lens 14 and G1 lens 13, and enters the second optical path dividing prism 10 from the second surface 10a of the second optical path dividing prism 10.

[0053] The first surface 10b of the second optical path splitting prism has a dichroic film that reflects infrared light. As shown in the reflected light path 31b, the eye image entering the second optical path splitting prism 10 is reflected from the first surface 10b toward the second surface 10a.

[0054] Then, as shown in imaging optical path 31c, the reflected eye image is totally internally reflected by the second surface 10a, exits the second optical path splitting prism 10 through the third surface 10c of the second optical path splitting prism 10, passes through the aperture 28, and is formed on the gaze detection sensor 30 via the gaze imaging lens 29. In addition to this eye image, gaze detection also uses a corneal reflection image, which is formed by specularly reflecting light from an infrared LED onto the cornea.

[0055] Figure 5 An example is shown of the light path that is reflected by the specular surface of the cornea 37 of the eyeball and received by the gaze detection sensor 30 from infrared LEDs 18, 19, 23 and 25 for near-field illumination.

[0056] <Description of line-of-sight detection operation>

[0057] Now refer to Figure 6 , Figure 7A , Figure 7B and Figure 8 This describes a method for detecting line of sight. The following example uses infrared LEDs 26 and 27, but the same method can be used when using other infrared LEDs 18, 19, 22-25. Figure 6This is a schematic diagram of an optical system used for line-of-sight detection, illustrating the principle of the line-of-sight detection method. For example... Figure 6 As shown, infrared LEDs 26 and 27 illuminate infrared light towards the user's eyeball 140. A portion of the infrared light emitted from infrared LEDs 26 and 27 and reflected by the eyeball 140 forms an image near the gaze detection sensor 30 via the gaze imaging lens 29. Figure 6 In this design, the positions of infrared LEDs 26 and 27, the gaze imaging lens 29, and the gaze detection sensor 30 are adjusted to facilitate understanding the principle of gaze detection.

[0058] Figure 7A This is a schematic diagram of an eye image (an eye image projected onto the eye detection sensor 30) captured by the gaze detection sensor 30. Figure 7B This is a graph showing the output intensity of the gaze detection sensor 30 (such as a CCD). Figure 8 A schematic flowchart of the gaze detection operation is shown.

[0059] When the gaze detection operation begins, Figure 8 In step S801, infrared LEDs 26 and 27 respond to the instruction from the light source driving unit 205 to emit infrared light toward the user's eyeball 140 with a light intensity E2 for gaze detection.

[0060] In step S802, the CPU 3 begins acquiring eye images via the gaze detection sensor 30. The pupil image and corneal reflection image of the user's eye, illuminated by infrared light, are formed near the gaze detection sensor 30 via the gaze imaging lens 29 (light receiving lens) and are photoelectrically converted by the gaze detection sensor 30. This obtains a processable electrical signal representing the eye image. Eye images are acquired continuously at regular intervals.

[0061] In step S803, the gaze detection unit 201 (gaze detection circuit) sends the eye image (eye image signal; electrical signal of eye image) obtained from the gaze detection sensor 30 to the CPU 3.

[0062] In step S804, CPU 3 obtains the coordinates of the point corresponding to the pupil center c from the eye image obtained in step S802.

[0063] In step S805, CPU 3 obtains the coordinates of two points corresponding to the corneal reflection images Pd and Pe of infrared LEDs 26 and 27.

[0064] Infrared light emitted from infrared LEDs 26 and 27 illuminates the cornea 142 of the user's eyeball 140. The corneal reflection images Pd and Pe, formed by a portion of the infrared light reflected from the surface of the cornea 142, are collected by the gaze imaging lens 29 and formed as corneal reflection images Pd' and Pe' in the eye image on the gaze detection sensor 30. Similarly, light from edge points a and b of the pupil 141 also forms pupil edge images a' and b' in the eye image on the gaze detection sensor 30.

[0065] Figure 7B Show Figure 7A The brightness information (brightness distribution) of region α' in the eye image. Figure 7B In the image, the horizontal direction of the eye image is the X-axis, and the vertical direction is the Y-axis, with the brightness distribution shown along the X-axis. In this embodiment, the X coordinates (coordinates along the X-axis (horizontal direction)) of the corneal reflection images Pd' and Pe' are Xd and Xe, and the X coordinates of the pupil edge images a' and b' are Xa and Xb. Figure 7B As shown, extremely high levels of brightness were obtained at the X coordinates Xd and Xe of the corneal reflectance images Pd' and Pe'. In the region corresponding to the pupil 141 (the region of the pupil image formed by light from the pupil 141 on the gaze detection sensor 30) from X coordinate Xa to X coordinate Xb, the brightness levels were extremely low except at X coordinates Xd and Xe. In the region of the iris 143 located outside the pupil 141 (the region of the iris image outside the pupil image formed by light from the iris 143), a brightness level between the two types of brightness was obtained. Specifically, a brightness level between the two types of brightness was obtained in regions where the X coordinate is less than X coordinate Xa and regions where the X coordinate is greater than X coordinate Xb.

[0066] From such Figure 7B The brightness distribution shown allows us to obtain the X coordinates Xd and Xe of the corneal reflection images Pd′ and Pe′, and the X coordinates Xa and Xb of the pupil edge images a′ and b′. Specifically, coordinates with extremely high brightness are obtained as the coordinates of the corneal reflection images Pd′ and Pe′, and coordinates with extremely low brightness are obtained as the coordinates of the pupil edge images a′ and b′. Furthermore, when the rotation angle θx of the optical axis of the eyeball 140 relative to the optical axis of the vision imaging lens 29 is small, the X coordinate Xc of the pupil center image c' (the center of the pupil image) formed on the vision detection sensor 30 by light from the pupil center c is expressed as Xc≈(Xa+Xb) / 2. That is, the X coordinate Xc of the pupil center image c' can be calculated based on the X coordinates Xa and Xb of the pupil edge images a' and b'. Thus, the X coordinates of the corneal reflection images Pd′ and Pe′ and the X coordinates of the pupil center image c′ are estimated.

[0067] CPU 3 also calculates the Y coordinates (coordinates in the Y-axis direction (vertical direction)) in the same way to obtain the coordinates (Xc, Yc) of the pupil center image c', the coordinates (Xd, Yd) of the corneal reflection image Pd', and the coordinates (Xe, Ye) of the corneal reflection image Pe'.

[0068] In step S806, CPU 3 calculates the imaging magnification β of the eye image. The imaging magnification β is a magnification determined using the position of the eyeball 140 relative to the line-of-sight imaging lens 29, and can be obtained as a function of the interval (Xd-Xe) between the corneal reflective images Pd' and Pe'.

[0069] In step S807, CPU 3 calculates the rotation angle of the optical axis of eyeball 140 relative to the optical axis of the viewing imaging lens 29. The X-coordinate of the midpoint between the corneal reflective images Pd and Pe is essentially equal to the X-coordinate of the center of curvature O of cornea 142. Thus, when the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, the rotation angle θx of eyeball 140 in the ZX plane (the plane perpendicular to the Y-axis) is obtained by the following expression 1. Furthermore, the rotation angle θy of eyeball 140 in the ZY plane (the plane perpendicular to the X-axis) can also be calculated using the same method as for the rotation angle θx.

[0070] β×Oc×SINθx≈{(Xd+Xe) / 2}–Xc…(Expression 1)

[0071] In step S808, the CPU 3 uses the rotation angles θx and θy calculated in step S807 to obtain (estimate) the user's gaze point (viewpoint; the position where the eyes are focused, the position the user is looking at) in the visual perception image displayed on the display panel 6. Assuming that the coordinates (Hx, Hy) of the gaze point are the coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the gaze point can be calculated using the following expressions 4 and 5 (or expressions 4' and 5').

[0072] In step S809, CPU 3 stores the coordinates (Hx, Hy) of the gaze point in memory unit 4 and ends the gaze detection operation.

[0073] The accuracy of gaze detection decreases when the eye is far from a predetermined position (the optical axis of the eyepiece optical system 16 in this embodiment) or when the number of corneal reflection images in the eye image differs from the predetermined number (the number of infrared LEDs illuminating the light).

[0074] An example is now described where the number of corneal reflective images is less than a predetermined number. Before the infrared light emitted from infrared LEDs 26 and 27 and reflected on the surface of the cornea 142 forms an image on the gaze detection sensor 30, the user's eyelids or eyelashes may block the light. In this case, because no corneal reflective image corresponding to the blocked infrared light is formed, the number of corneal reflective images will be less than two. When the number of corneal reflective images does not exceed one, the imaging magnification β cannot be calculated, thus reducing the gaze detection accuracy. If the user opens their eyelids wide, so that the eyelids or eyelashes do not block the infrared light, this reduction in gaze detection accuracy can be avoided.

[0075] Now, an example is described where the number of corneal reflection images exceeds a predetermined number. When a user wears glasses, infrared light is reflected from either the incident (front) or exit (rear) surface of the lens, depending on the position and orientation of the glasses. This can produce artifacts in the eye image. If artifacts are incorrectly detected as corneal reflection images, the number of corneal reflection images (detection count) will be greater than two. As a result, incorrect gazes will be detected based on the coordinates of the artifacts (reduced gaze detection accuracy). This reduction in gaze detection accuracy can be avoided if the user adjusts the position and orientation of their glasses so that artifacts do not appear in the areas where they are incorrectly detected as corneal reflection images.

[0076] As described above, the accuracy of gaze detection can decrease due to various factors, and there are various methods to avoid this decrease in gaze detection accuracy (methods for adjusting the viewing state perceived by the display panel 6). Therefore, this embodiment advantageously notifies the method for adjusting the viewing state. Notifications can be given in various ways, such as through display, audio, or a combination of display and audio, and this embodiment is an example of giving a notification through display on the display panel 6.

[0077] <Description of calibration operation>

[0078] Reference Figure 9 as well as Figures 10A to 10J This describes the calibration and notification procedures. Figure 9 A schematic flowchart of the calibration operation is shown. Figures 10A to 10J Examples of displayed images and eye images are shown. Gaze patterns vary between individuals, and these individual differences need to be considered for accurate gaze detection. Therefore, a calibration operation is required to account for individual differences. The calibration operation is used to obtain corrected values ​​based on the user's individual gaze characteristics. Figure 8 A calibration operation is performed before the gaze detection operation. The notification operation is the operation that notifies the user of the method for adjusting the viewing state. In this embodiment, the notification operation is performed during the calibration operation to encourage the user to... Figure 8Before the gaze detection operation, one must be accustomed to how to use camera 1 (how to look at display panel 6; how to observe the viewfinder). Nevertheless, it is possible to... Figure 8 Notifications are sent during the line-of-sight detection operation.

[0079] When the calibration operation begins, Figure 9 In step S901, CPU 3 displays the target or instruction for calibration on display panel 6. For example, such as... Figure 10A As shown, the display shows the target 1002 to be viewed (focused on) by the user, and the instruction 1003 informing the user that they should look at the target 1002 and the ideal viewing state (viewing method). The display area 1001 is the display area (area on the display surface) of the display panel 6.

[0080] Steps S902 to S905 and Figure 8 Steps S801 to S804 are the same. In step S905, when there are no coordinates with extremely low brightness in the eye image, the CPU 3 determines that it is impossible to detect (calculate) the coordinates (Xc, Yc) of the pupil center image c′ (the center of the pupil image).

[0081] In step S906, CPU 3 determines whether a pupil image has been captured in the eye image; specifically, it determines whether the coordinates (Xc, Yc) of the pupil center image c' can be detected. If it is determined that a pupil image has been captured (and the coordinates (Xc, Yc) can be detected), the process proceeds to step S908. Otherwise, the process proceeds to step S907.

[0082] In step S907, CPU 3 updates the display on display panel 6 to notify the eye that it should move to a predetermined position. For example, as Figure 10B As shown, the following instruction 1004 is displayed: instructing the eye to translate towards the central axis of the display panel 6 (perpendicular to the central axis of the display surface), or towards the optical axis of the eyepiece optical system 16, i.e., the center of the viewfinder. In this case, as... Figure 10C As shown, for example, because the pupil image is located outside the field of view 1005 (camera area) of the gaze detection sensor 30, the pupil image is not shown in the eye image. In this case, since it is difficult to determine the position of the pupil image, the CPU 3 only notifies the target position of the eye, but not the direction of eye movement.

[0083] In step S908, CPU 3 determines whether a pupil image has been captured within a predetermined region of the eye image. Specifically, it determines whether the coordinates (Xc, Yc) of the pupil center image c' are detected within the predetermined region. If it is determined that a pupil image has been captured within the predetermined region (coordinates (Xc, Yc) are detected within the predetermined region), the process proceeds to step S910. Otherwise, or if it is determined that a pupil image has not been captured within the predetermined region (coordinates (Xc, Yc) are not detected within the predetermined region; a pupil image has been captured outside the predetermined region; coordinates (Xc, Yc) are detected outside the predetermined region), the process proceeds to step S909.

[0084] In this embodiment, such as Figure 10D As shown, the field of view 1005 of the gaze detection sensor 30 is divided into nine regions A1 to A9, using thresholds Xc1 and Xc2 for the X coordinates and thresholds Yc1 and Yc2 for the Y coordinates as boundaries. In step S908, the CPU 3 determines whether Xc1≤Xc≤Xc2 and Yc1≤Yc≤Yc2. Then, if Xc1≤Xc≤Xc2 and Yc1≤Yc≤Yc2, that is, if coordinates (Xc, Yc) are detected in the central region A5, the process proceeds to step S910. If Xc1≤Xc≤Xc2 and Yc1≤Yc≤Yc2 are not satisfied, that is, if coordinates (Xc, Yc) are detected in one of the regions A1 to A4 and A6 to A9 other than region A5, the process proceeds to step S909.

[0085] Even when the user maintains the same facial position and orientation (posture) with their eyes centered in the viewfinder, eye movement will shift the pupil image and change the coordinates (Xc, Yc). Since the changed coordinates (Xc, Yc) still correspond to the appropriate viewing position, it is not desirable to prompt the user to move their eyes. Therefore, it is preferable to determine thresholds Xc1, Xc2, Yc1, and Yc2 such that the region including the changed coordinates (Xc, Yc) is set as region A5.

[0086] In step S909, as in step S907, CPU 3 updates the display on display panel 6 to notify the eye that it should move to a predetermined position. Here, CPU 3 can determine the direction from the pupil image toward the predetermined area based on the coordinates (Xc, Yc) of the pupil center image c'. Specifically, CPU 3 determines the direction from the area with detected coordinates (Xc, Yc) in areas A1 to A4 and A6 to A9 toward area A5. CPU 3 can also determine the direction of eye movement toward the predetermined position (the center of the viewfinder) corresponding to the determined direction. Therefore, CPU 3 updates the display on display panel 6 to also indicate the determined direction of movement. For example, as Figure 10EAs shown, the display includes an instruction 1006 instructing the eye to translate along the determined direction of movement, and an arrow 1007 indicating the determined direction of movement. Figure 10D In the state where coordinates (Xc, Yc) are detected in region A7, such as Figure 10E As shown, the instruction 1006 and arrow 1007 are used to move the eye to the upper right.

[0087] Step S910 and Figure 8 The same applies to step S805. In step S910, the CPU 3 also counts the number of corneal reflection images (detection count) in the eye image.

[0088] In step S911, CPU 3 compares the number of corneal reflection images (detection count) in the eye image with a predetermined count. The predetermined count is the number of emitting infrared LEDs (the number of infrared light rays). When only infrared LEDs 26 and 27 are emitting light, the predetermined count is 2. If it is determined that the detection count is equal to the number of irradiated light rays, the process proceeds to step S916. If it is determined that the detection count is greater than the number of irradiated light rays, the process proceeds to step S912. If it is determined that the detection count is less than the number of irradiated light rays, the process proceeds to step S915.

[0089] In step S912, CPU 3 updates the display on display panel 6 to indicate that the glasses should be moved to identify the artifact (false corneal reflection image).

[0090] Here, the number of corneal reflection images detected is greater than the predetermined number (the number of infrared illumination rays). Figure 10F This example shows an eye image where the number of corneal reflective images detected is greater than the number of infrared light rays illuminating the eye. Figure 10F In this context, images 1008a and 1009a are corneal reflection images formed on the gaze detection sensor 30 by infrared light emitted from infrared LEDs 26 and 27 and reflected only on the corneal surface of the eye, not on the user's eyeglass lenses. Therefore, images 1008a and 1009a are necessary for gaze detection. Images 1008b and 1009b are artifacts formed on the gaze detection sensor 30 by infrared light emitted from infrared LEDs 26 and 27 and reflected on the user's eyeglass lenses. Therefore, images 1008b and 1009b are not necessary for gaze detection. Artifacts 1008b and 1009b, which are similar in size and brightness to corneal reflection images 1008a and 1009a, may be mistakenly detected as corneal reflection images. When such false detections occur, even with detailed analysis of individual eye images, it is not easy to identify (distinguish) the artifacts in images 1008a, 1009a, 1008b, and 1009b that are detected as corneal reflection images.

[0091] Therefore, in step S912, as Figure 10G As shown, CPU 3 displays the following instruction 1010: It indicates that only the glasses should be moved, without moving the eyes (only the position and orientation of the glasses should be changed).

[0092] In step S913, the CPU 3 identifies (distinguishes) artifacts in the detected corneal reflection image based on multiple eye images. Figure 10H Showing the user's response Figure 10G Instruction 1010 provides an example of an eye image obtained only after moving the glasses. (See example...) Figure 10H As shown, when only the glasses are moved, the non-artifact corneal reflection image moves a small distance, while the artifact image moves a large distance. Therefore, artifacts are identified by detecting the movement of the detected corneal reflection image from multiple eye images.

[0093] Specifically, CPU 3 uses feature tracking, a known image parsing technique, to determine the correspondence between the eye image obtained before moving the glasses and the eye image obtained after moving the glasses (images detected as corneal reflection images). Here, it is assumed that the eye image obtained before moving the glasses is... Figure 10F The eye image obtained after moving the glasses is Figure 10H The eye image. In this case, it is judged as Figure 10H Image 1008a' and Figure 10F Image 1008a corresponds to image 1009a', image 1008b' corresponds to image 1008b, and image 1009b' corresponds to image 1009b.

[0094] Based on the coordinates of the images before and after the glasses were moved, CPU 3 calculates the amount of movement (distance) of each image caused by the movement of the glasses.

[0095] Then, CPU 3 identifies images 1008a' and 1009a' with a movement distance not greater than a predetermined distance as corneal reflection images, and identifies images 1008b' and 1009b' with a movement distance greater than the predetermined distance as artifacts.

[0096] In step S914, CPU 3 updates the display on display panel 6 to indicate that the glasses should be moved to remove the artifact. For example, as Figure 10I As shown, items 1008b" and 1009b" correspond to the identified artifacts 1008b' and 1009b', respectively, and item 1005" corresponds to the region of the eye image. The positions of the regions of items 1008b" and 1009b" relative to item 1005" are shown relative to the regions of artifacts 1008b' and 1009b'. Figure 10HThe position corresponds to the visual field area 1005. When artifacts 1008b' and 1009b' move, items 1008b" and 1009b" also move. Therefore, when the glasses are moved, causing items 1008b" and 1009b" to move out of the area of ​​item 1005", artifacts 1008b' and 1009b' disappear, and consequently items 1008b" and 1009b" disappear. For this reason, as... Figure 10I As shown, the following instruction 1011 is also displayed: It is indicated that the glasses should be moved to move items 1008b" and 1009b" out of the area of ​​item 1005".

[0097] In step S915, CPU 3 updates the display on display panel 6 to indicate that the eyelids should be widened. For example, as Figure 10J As shown, display instruction 1012 and graphic 1013 indicate that the opening of the eyelids should be widened. Normally, closing the eyelid of one eye reduces the opening of the eyelid of the other eye. Therefore, when the eyelid of the eye not observing the viewfinder is closed, the opening of the eyelid of the eye observing the viewfinder tends to decrease. For this reason, as... Figure 10J As shown, display instruction 1012 informs that the eyelids of both eyes should be opened wider (the eyelids of both eyes should be opened). Additionally, display graphic 1014 is used for this notification.

[0098] Steps S916 and S917 and Figure 8 Steps S806 and S807 are the same.

[0099] In step S918, the CPU 3 uses the rotation angles θx and θy calculated in step S917 to obtain (estimate) the user's gaze point in the visual perception image displayed on the display panel 6. This step obtains a gaze point that does not reflect the result (correction value) of the calibration operation, i.e., a gaze point that does not reflect the personal characteristics of the user's gaze. The coordinates (H'x, H'y) of this gaze point can be calculated using the following expressions 2 and 3.

[0100] H'x = m × θx…(Expression 2)

[0101] H'y = m × θy…(Expression 3)

[0102] The parameter m in expressions 2 and 3 is a constant determined by the configuration of the viewfinder optical system of camera 1 (e.g., the line-of-sight imaging lens 29). Parameter m is a transformation coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c in the image for visual perception. Parameter m is predetermined and stored in memory unit 4.

[0103] In step S919, CPU 3 obtains the correction values ​​Bx and By and uses the coordinates (H'x, H'y) calculated in step S918.

[0104] The coordinates (Hx, Hy) of the line-of-sight point that reflects the result of the calibration operation (correction value Bx, By) are expressed by the following expressions 4 and 5.

[0105] Hx = m × (θx + Bx) … (Expression 4)

[0106] Hy = m × (θy + By) … (Expression 5)

[0107] Here, users should pay attention to, for example Figure 10A The target 1002 is shown, and the target 1002 is displayed in the center of the display area 1001. Therefore, the coordinates (Hx, Hy) of the correct line of sight are (0, 0). Thus, based on the coordinates (Hx, Hy) = (0, 0) and expressions 2 to 5, it is possible to obtain the correction value Bx equal to -H'x / m and the correction value By equal to -H'y / m.

[0108] In step S920, CPU 3 stores the correction values ​​Bx and By obtained in step S918 in memory unit 4 and ends the calibration operation.

[0109] In the aforementioned example, a target 1002 displayed in the center of the display area 1001 is used as the target to be viewed by the user during the calibration operation. However, the position of the target to be viewed by the user is not limited to the center of the display area 1001. For example, the target to be viewed by the user can be displayed at a position offset from the center of the display area 1001 in any direction above, below, to the left, and to the right.

[0110] Multiple targets to be viewed by the user can be displayed sequentially or together, allowing the user to view multiple locations in turn. In this case, for example, expressions 4 and 5 used in steps S808 and S919 can be replaced by expressions 4' and 5' as follows.

[0111] Hx = m × (Ax × θx + Bx) … (Expression 4')

[0112] Hy = m × (Ay × θy + By) … (Expression 5')

[0113] In this scenario, in step S918, the CPU 3 calculates the coordinates (H'x, H'y) of the gaze point for each of the multiple targets to be viewed by the user. In step S919, the CPU 3 calculates correction values ​​Ax, Ay, Bx, and By for each target, minimizing the error in the coordinates (Hx, Hy) of the gaze point (the difference between the coordinates (Hx, Hy) and the correct coordinates). In step S920, the CPU 3 stores the correction values ​​Ax, Ay, Bx, and By in the memory unit 4. This calibration operation allows for high accuracy in gaze detection.

[0114] As described above, this embodiment provides a notification of a method for adjusting the viewing state of the visually perceived display unit based on at least one of the position of the pupil image in the eye image and the number of corneal reflection images in the eye image (detection count). As a result, the user can easily adjust the position of the eyes (position and orientation of the face), the position of the glasses, the orientation of the glasses, etc., thus enabling gaze detection with high accuracy.

[0115] (Variant Example)

[0116] In the above embodiments, an example has been given of a configuration where the line of sight of a user looking directly at the front is aligned with the optical axis of the optical system used to detect the line of sight; however, the present invention is not limited to this. For example, as a line of sight detection method, a so-called off-axis line of sight detection method (method) can be adopted, in which the line of sight of a user looking directly at the front is different from the camera axis of the eyeball image associated with line of sight detection. That is, a method can be adopted in which the user's eyeball is directly detected by the line of sight detection unit without passing through another optical system. In this case, the configuration can be such that, without passing through the eyepiece optical system 16 described above, the line of sight detection sensor that captures the image of the user's eyeball is positioned outside the optical axis of the eyepiece optical system 16, and the image of the user's eyeball looking at the display panel 6 is captured from an angle relative to the optical axis.

[0117] In the following text, a variation of the method of directly detecting the user's gaze without employing the optical path splitting prism unit 11 will be described. Since the basic configuration and basic driving method of the camera 1, which is the imaging device according to this variation, are basically the same as in the above embodiment, the reference numerals for each unit are the same, and their descriptions will be omitted. In this variation, a configuration different from the above embodiment will be described in detail later.

[0118] Figure 11 This is an example of a cross-sectional view of camera 1 according to this variant. Figure 11 In this embodiment, a cover glass 1101 is disposed on the display panel unit 8. That is, the difference between this variant and the above embodiment is that the optical path dividing prism unit 11 (optical path dividing member) is removed from the interior of the camera 1, and a new cover glass 1101 is provided. The cover glass 1101 is a transparent glass component, and compared to the optical path dividing prism 11 in the above embodiment, the thickness of the cover glass 1101 in the optical axis direction of the EVF portion can be reduced. Therefore, in the camera 1 according to this variant, by bringing the eyepiece optical system 16 closer to the display panel unit 8, the magnification of the eyepiece optical system 16 can be increased, thereby improving optical performance.

[0119] In addition, such as Figure 11As shown, the gaze detection module 1102, formed by the gaze imaging lens 1102a and the gaze detection sensor 1102b, is arranged in parallel with the infrared LED 18. The gaze detection module 1102 is arranged at a predetermined angle θ relative to the optical axis of the eyepiece optics system 16 (i.e., the optical axis of the EVF portion), and is capable of directly capturing the photographer's eyeball through the illumination window 20. In this variant, it is assumed that the angle formed by the optical axis of the gaze imaging lens 1102a and the optical axis of the eyepiece optics system 16 is the predetermined angle θ.

[0120] Here, since the gaze detection module 1102 in this variant captures the subject's eyeball without using the eyepiece optical system 16, the size and position of the eyeball image projected onto the gaze detection sensor 1102b change according to the distance between the gaze detection module 1102 and the photographer's eyeball.

[0121] Furthermore, as in this variant, when the gaze detection module 1102 is located below the photographer's eyeball, the magnification of the eyeball image formed on the gaze detection sensor 1102b changes according to the distance from the photographer's eyeball, while the eyeball image moves in the vertical direction. The side of the camera 1 where the EVF portion is arranged is the upper side. Therefore, compared to the above embodiment, the imaging area 1201 of the gaze detection sensor 1102b in this variant ( Figure 12A and Figure 12B The field of view is expanded in the vertical direction. If the gaze detection module 1102 is positioned to the left or right of the photographer's eye, the field of view is expected to expand in the horizontal direction because the eye image moves in the horizontal direction according to the distance from the photographer's eye.

[0122] Figure 12A and Figure 12B This is a diagram showing an image of the photographer's eyeball formed on the gaze detection sensor 1102b according to this variant. Figure 12A This shows the situation where the photographer's eyeball is close to the gaze detection sensor 1102b. Figure 12B This illustrates a situation where the photographer's eye is far from the line-of-sight detection sensor 1102b. That is, as... Figure 12A and Figure 12B As shown, the ratio of the eye image to the detected eye image and the position of the eye image in the eye image are also changed according to the distance between the gaze detection sensor 1102b and the photographer (eyeball).

[0123] Therefore, in this variant example, as Figure 12A and Figure 12BAs shown, depending on the distance between the gaze detection sensor 1102b and the photographer (eyeball), the area on the gaze detection sensor 1102b that detects the eyeball changes to area 1202 or area 1203. That is, in this variant, the presence or absence of an eyeball image is detected within the aforementioned area, and various notifications are displayed to the user as in the above embodiment. The distance between the gaze detection sensor 1102b and the photographer can be calculated based on the imaging magnification β calculated using a corneal reflection image, as in the above embodiment, or it can be estimated based on the size of the iris portion in the eyeball image.

[0124] The above embodiments (including variations) are merely examples, and the present invention also includes configurations obtained by appropriately modifying or changing the above configurations within the scope of the present invention. The present invention also includes configurations obtained by appropriately combining the above configurations. For example, in the above embodiments (including variations), a configuration in which the gaze detection unit is disposed in the EVF portion of the camera device has been described, but the present invention is not limited thereto. An electronic device provided with (or combined with) the above gaze detection unit can be any device such as a head-mounted display.

[0125] This disclosure allows users to easily adjust the position of the face, the orientation of the face, the position of the glasses, the orientation of the glasses, etc., thus enabling gaze detection with high accuracy.

[0126] <Other Implementation Methods>

[0127] The embodiments of the present invention can also be implemented by the following method: providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reading out and executing the program.

[0128] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the above claims is accorded the broadest description to encompass all such variations, equivalent structures, and functions.

Claims

1. An electronic device capable of acquiring an eye image of an eye looking at a display unit, said electronic device comprising: A detection unit configured to perform gaze detection based on the eye image; The judgment unit is configured to determine whether a predetermined condition is met, the predetermined condition including whether a pupil image is captured in a predetermined area in the eye image and whether the number of corneal reflection images in the eye image is less than at least one of a predetermined number; as well as A control unit is configured to control whether to issue a predetermined notification related to a method for adjusting a viewing state, based on the result of a judgment by the judgment unit, wherein the display unit is visually perceived in the viewing state.

2. The electronic device according to claim 1, wherein, The determination unit determines whether the predetermined condition is met based on the eye image obtained during the calibration operation related to the gaze detection.

3. The electronic device according to claim 1, wherein, If no pupil image is captured within the predetermined area in the eye image, the control unit performs control to provide a notification to move the eye to a predetermined position corresponding to the predetermined area as the predetermined notification.

4. The electronic device according to claim 3, wherein, If the pupil image is captured outside the predetermined area, the control unit performs control to further provide a notification of the direction of movement of the eye toward the predetermined position as the predetermined notification, wherein the direction of movement corresponds to the direction from the pupil image toward the predetermined area.

5. The electronic device according to claim 4, wherein, The notification of the direction of movement includes displaying an arrow on the display unit.

6. The electronic device according to claim 1, wherein, The determining unit determines whether the number of corneal reflection images is less than the predetermined number, and The control unit performs control based on the result of the judgment, causing the predetermined notification to change its content.

7. The electronic device according to claim 6, wherein, If the number of corneal reflective images is less than the predetermined number, the control unit performs control to provide a notification related to the degree of eyelid opening as the predetermined notification.

8. The electronic device according to claim 7, wherein, The content related to the degree of eyelid opening is related to widening the opening of the eyelids.

9. The electronic device according to claim 6, wherein, If the number of corneal reflective images exceeds the predetermined number, the control unit performs control to provide a notification related to the movement of the glasses as the predetermined notification.

10. The electronic device according to claim 9, wherein, After controlling the movement of the glasses to provide notification, the control unit: Control is performed to display on the display unit a first item corresponding to a corneal reflection image that has moved a distance greater than a predetermined distance, and a second item corresponding to a region of the eye image. Control is exercised to provide a notification to move the glasses to remove the first item from the area of ​​the second item.

11. The electronic device according to claim 2, wherein, The status during a calibration operation related to line-of-sight detection is the status of the items used for the calibration displayed on the display unit.

12. The electronic device according to claim 1, further comprising a gaze detection sensor, wherein, The gaze detection sensor is capable of obtaining the eye image based on an optical image incident through an optical system used for the display unit, wherein the optical system is disposed between the display unit and the eyepiece.

13. The electronic device according to claim 1, further comprising a gaze detection sensor, wherein, The gaze detection sensor is capable of obtaining the eye image based on an incident optical image that does not pass through the optical system used for the display unit, wherein the optical system is disposed between the display unit and the eyepiece. The detection unit adjusts the range of the eye image used for eye detection based on information related to the distance between the eye detection sensor and the photographer.

14. A control method for an electronic device, capable of acquiring an eye image of an eye looking at a display unit, the control method comprising: Gaze detection is performed based on the eye image; Determine whether a predetermined condition is met, the predetermined condition including whether a pupil image is captured in a predetermined area in the eye image and whether the number of corneal reflection images in the eye image is less than at least one of a predetermined number; as well as Based on the result of the judgment, control whether to issue a predetermined notification related to the method of adjusting the viewing state, wherein the display unit is visually perceived in the viewing state.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the control method of claim 14.

16. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the control method of claim 14.

Images