Electronic apparatus and electronic apparatus control method

The electronic device improves image visibility and user operability by dynamically controlling line-of-sight and AF frame displays based on gaze estimation and imaging conditions, addressing the inconsistency in existing technologies.

JP2024170166APending Publication Date: 2024-12-06CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023087182
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-12-06

Smart Images

  • Figure 2024170166000001_ABST
    Figure 2024170166000001_ABST
Patent Text Reader

Abstract

To provide an electronic apparatus which achieves both displaying of a visual line position and securing visibility of an image.SOLUTION: The electronic apparatus comprises: acquisition means which acquires information on the visual line position of a user viewing a captured image; estimation means which on the basis of the visual line position of the user, estimates a subject which the visual line of the user is oriented to; and control means which on the basis of the situation of imaging of the image and / or the situation of the subject included in the image, controls whether or not to display a first item indicating the subject estimated by the estimation means.SELECTED DRAWING: Figure 8A
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an electronic device and a method for controlling an electronic device. [Background technology]

[0002] Conventionally, imaging devices equipped with a function for detecting the gaze position of a user looking into the viewfinder field of view have been provided. In recent years, imaging devices with electronic viewfinders capable of displaying images acquired by an imaging element that receives a light beam that has passed through a photographing optical system have been provided. Imaging devices with electronic viewfinders are capable of recognizing and detecting various types of subjects. Such imaging devices are capable of performing focus detection control at a subject position that reflects the user's intention, based on information on the detected gaze position and the recognition result of the subject in the acquired image.

[0003] Patent Document 1 discloses a technique for appropriately controlling the display state of the gaze position depending on the status of the autofocus (hereinafter referred to as AF) function, in cases where it is desired to prioritize gaze detection and cases where it is desired to prioritize display in the viewfinder. Patent Document 2 discloses a technique for appropriately setting a predetermined time for determining when a user's gaze stops in the viewfinder when a corresponding function is activated when the user's gaze stops for a predetermined period of time or more. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-131441 A [Patent Document 2] Japanese Patent Application Publication No. 09-015483 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in an actual shooting scene, it is preferable that the display of the gaze position and the AF frame, etc. be appropriately controlled in consideration of visibility and user operability, etc. For example, when gaze position information is used to determine the AF target area (AF frame), even if the gaze retention time until the AF frame is displayed is set in advance, the display state of the AF frame is not necessarily changed in a timely manner for the user.

[0006] Therefore, an object of the present invention is to provide an electronic device that can display the gaze position while ensuring the visibility of an image. [Means for solving the problem]

[0007] A first aspect of the present invention is an electronic device characterized by having an acquisition means for acquiring information on the gaze position of a user viewing a captured image, an estimation means for estimating a subject at which the user's gaze is directed based on the user's gaze position, and a control means for controlling whether to display a first item indicating the subject estimated by the estimation means based on at least one of the capture conditions of the image and the conditions of the subject included in the image.

[0008] A second aspect of the present invention is a control method for an electronic device, characterized by comprising an acquisition step of acquiring information on the gaze position of a user viewing a captured image, an estimation step of estimating a subject at which the user's gaze is directed based on the user's gaze position, and a control step of controlling whether to display a first item indicating the subject estimated in the estimation step based on at least one of the capture conditions of the image and the conditions of the subject included in the image.

[0009] A third aspect of the present invention is a program for causing a computer to function as each of the means of the electronic device described above.A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device described above. Effect of the Invention

[0010] According to the present invention, it is possible to provide an electronic device that can display the gaze position while ensuring the visibility of an image. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the appearance of a camera. [Diagram 2] FIG. 2 is a diagram illustrating an example of the internal configuration of a camera. [Diagram 3] FIG. 2 is a block diagram showing an example of an electrical configuration within the camera. [Figure 4] FIG. 4 is a diagram showing an example of a display in a viewfinder field of view. [Diagram 5] 1 is a diagram for explaining the principle of a gaze detection method. [Figure 6] FIG. 2 shows an eye image. [Figure 7] 13 is a flowchart of a gaze detection process. [Figure 8A] 11 is a flowchart illustrating an icon display control process. [Figure 8B] 13 is a flowchart of an icon display determination process based on an imaging situation. [Figure 8C] 13 is a flowchart of an icon display determination process based on a subject situation. [Figure 9] FIG. 4 is a diagram illustrating subject movement information. [Figure 10] FIG. 13 is a diagram illustrating an icon indicating a gaze position. [Figure 11] FIG. 11 is a diagram illustrating icon display based on a zoom operation. [Figure 12] 11A and 11B are diagrams illustrating icon display based on changes in camera attitude. [Figure 13] 11A and 11B are diagrams illustrating icon display based on a change in the number of subjects. [Figure 14] 11A and 11B are diagrams illustrating icon display based on subject movement information. [Figure 15] 11 is a flowchart of an icon display determination process according to the second embodiment. [Figure 16] 11A and 11B are diagrams illustrating icon display in a focus locked state. [Figure 17] FIG. 11 is a diagram illustrating icon display in a focus tracking state. [Figure 18] FIG. 13 is an external view of another electronic device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Embodiment 1] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0013] <Configuration Description> An example of the camera configuration will be described with reference to Figs. 1(a), 1(b) to 3. Figs. 1(a) and 1(b) show an example of the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to an embodiment. Fig. 1(a) is a front perspective view, and Fig. 1(b) is a rear perspective view. As shown in Fig. 1(a), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operating member capable of receiving a photographing operation from a user (photographer), is disposed on the camera housing 1B.

[0014] As shown in FIG. 1(b), an eyepiece lens 12 (eyepiece optical system) is disposed on the rear surface of camera housing 1B, through which the user looks to view a display device 10 (display panel) contained within camera housing 1B. The eyepiece optical system may include multiple lenses. Operation members 41 to 43 capable of receiving various operations from the user are also disposed on the rear surface of camera housing 1B. For example, operation member 41 is a touch panel capable of receiving touch operations, operation member 42 is an operation lever that can be pushed down in each direction, and operation member 43 is It is a four-way key that can be pressed in each of the four directions. The operation member 41 (touch panel) is provided with a display panel such as a liquid crystal panel, and has a function of displaying images on the display panel.

[0015] Fig. 2 is a cross-sectional view of camera 1 cut along the YZ plane defined by the Y axis and Z axis shown in Fig. 1(a), and shows an example of the internal configuration of camera 1. In Fig. 2, the same components as those in Figs. 1(a) and 1(b) are denoted by the same reference numerals.

[0016] Photographing lens unit 1A includes two lenses 101, 102, an aperture 111, an aperture drive section 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. Photographing lens unit 1A also includes an acceleration sensor 119, a zoom drive section 120, and a lens CPU 121. Note that for simplicity, two lenses 101, 102 are shown, but photographing lens unit 1A actually includes more than two lenses. Each component included in photographing lens unit 1A is controlled by lens CPU 121.

[0017] Aperture 111 provided in photographing lens unit 1A is driven by aperture drive section 112. Lens drive member 114 includes a drive gear and the like. Photocoupler 115 detects the rotation of pulse plate 116 linked to lens drive member 114 and transmits it to focus adjustment circuit 118. Focus adjustment circuit 118 drives lens drive motor 113 based on information from photocoupler 115 and information from camera housing 1B (information on lens drive amount), and moves lens 101 to change the focus position. Mount contact 117 is an interface between photographing lens unit 1A and camera housing 1B.

[0018] Acceleration sensor 119 is built into photographing lens unit 1A and detects panning of lenses 101, 102. Acceleration sensor 6 is built into camera housing 1B and detects panning of camera housing 1B. Panning of camera 1 is determined using at least one of acceleration sensor 119 built into photographing lens unit 1A and acceleration sensor 6 built into camera housing 1B.

[0019] The zoom driving unit 120 performs zoom driving based on a user's manual operation of an operating member provided on a lens barrel (not shown), or performs zoom driving based on an instruction from the camera housing 1B notified via the mount contact 117. The lens CPU 121 notifies the camera housing 1B of zoom driving information from the photographing lens unit 1A via the mount contact 117. The camera 1 can detect a change in the angle of view due to the zoom operation based on the zoom driving information.

[0020] The camera housing 1B includes an imaging element 2, a CPU 3, a memory section 4, a display device 10, a display device drive circuit 11, and the like. The imaging element 2 is disposed at a planned imaging surface of the photographing lens unit 1A. The CPU 3 is a central processing section of a microcomputer, and controls the entire camera 1. The memory section 4 stores images captured by the imaging element 2, and the like. The display device 10 is composed of a liquid crystal or the like, and displays the captured image (subject image), and the like, on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. A user can view an image (such as an image captured by the imaging element 2) displayed on the display surface of the display device 10 through an eyepiece 12.

[0021] The camera housing 1B also includes an eyepiece lens 12, light sources 13a and 13b, a light splitter 15, a light receiving lens 16, and an ocular imaging element 17. The light sources 13a and 13b are light sources for illuminating a user's eyeball 14. The light sources 13a and 13b detect the line of sight (the direction of the line of sight; the direction the user is looking) based on the relationship between the reflected image of light due to the corneal reflection (corneal reflection image; Purkinje image) and the pupil. Specifically, each of light source 13a and light source 13b is an infrared light emitting diode or the like that emits infrared light that is insensitive to the user, and is disposed around eyepiece lens 12.

[0022] The illuminated optical image of eyeball 14 (eye image; an image formed by light emitted from light sources 13a, 13b and reflected by eyeball 14) passes through eyepiece lens 12 and is reflected by light splitter 15. The eye image is then formed by light receiving lens 16 on ocular imaging element 17, which is a two-dimensional array of photoelectric elements such as CCD or CMOS. Light receiving lens 16 positions the pupil of eyeball 14 and ocular imaging element 17 in a conjugate imaging relationship. The line of sight of eyeball 14 is detected by a predetermined algorithm from the positional relationship between the pupil and the corneal reflection image in the eye image formed on ocular imaging element 17.

[0023] Fig. 3 is a block diagram showing an example of an electrical configuration in the camera 1. In Fig. 3, the same components as those in Figs. 1(a), 1(b) and 2 are denoted with the same reference numerals, and detailed description of these components will be omitted. The CPU 3 is connected to a line of sight detection circuit 201, a photometry circuit 202, an auto focus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, an acceleration sensor 6, a memory unit 4, an image processing unit 208 and an attitude detection unit 207.

[0024] Furthermore, CPU 3 transmits a signal to lens CPU 121 via mount contact 117. Lens CPU 121 controls focus adjustment circuit 118 arranged in photographing lens unit 1A, aperture control circuit 206 included in aperture drive section 112, acceleration sensor 119, and zoom drive section 120 based on the signal from CPU 3.

[0025] The memory unit 4 associated with the CPU 3 has a function of storing image signals from the image sensor 2 and the eye image sensor 17. The CPU 3 converts the image signal of the image sensor 2 stored in the memory unit 4 into a photographed image, and transfers the converted photographed image to a recording medium 7 connected to the memory unit 4.

[0026] The attitude detection unit 207 detects attitude information of the user holding the camera 1. The attitude detection unit 207 can detect the attitude information of the user using, for example, the acceleration sensor 6. The camera 1 can capture images in a normal position (horizontal position) in which the image sensor 2 is horizontal to the subject, and in a vertical position in which the image sensor 2 is vertical to the subject. The attitude detection unit 207 can detect a change in the attitude of the camera 1 even if the user changes the attitude of the camera 1 while looking through the viewfinder.

[0027] The gaze detection circuit 201 A / D converts the output of the eye imaging element 17 (an image of the eye) when an eyeball image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points used for gaze detection from the eye image according to a predetermined algorithm, and calculates the user's viewpoint (gaze position; position where the gaze is directed; position where the user is looking) on ​​the display surface of the display device 10 from the positions of the feature points.

[0028] The photometry circuit 202 amplifies, logarithmically compresses, and A / D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor, specifically the luminance signal corresponding to the brightness of the field, and sends the resulting field luminance information to the CPU 3. The field luminance information is information indicating the luminance of each pixel of the captured image. The photometry circuit 202 also detects a specific subject (specific object) from the image captured by the image sensor 2, and sends the resulting subject information and subject area information to the CPU 3. The subject information (object information) is information related to the detected subject, and the subject area information (object area information) is information indicating the area of ​​the detected subject. The subject information and subject area information can also be obtained by analyzing the field luminance information.

[0029] The auto focus detection circuit 203 A / D converts signal voltages from a plurality of detection elements (a plurality of pixels) included in the image sensor 2 and used for phase difference detection, and sends the converted signal to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the plurality of detection elements. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are focus detection points at each of 180 locations on the image plane corresponding to the field of view in the viewfinder (the field of view when looking through the viewfinder) shown in FIG. 4(a), specifically, the 180 locations shown on the display surface of the display device 10.

[0030] Switches SW1 and SW2 are connected to the signal input circuit 204. Switch SW1 is turned on by the first stroke (half press) of the release button 5, and is a switch for starting the photometry and focus detection (distance measurement) of the camera 1. Switch SW2 is turned on by the second stroke (full press) of the release button 5, and is a switch for starting the photographing operation. ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3. Photometry and focus detection are performed within a set photometry area or focus detection area, or within the area of ​​a selected (determined) main subject. For example, in photometry, the photometry circuit 202 determines the luminance of the target area from field luminance information, and in focus detection, the auto focus detection circuit 203 determines the distance to a subject present in the target area.

[0031] An operation member 41 (touch panel), an operation member 42 (operation lever), and an operation member 43 (four-way key) send operation signals corresponding to operations received from a user to the CPU 3. A light source drive circuit 205 drives the light sources 13a and 13b.

[0032] The image processing unit 208 applies various types of image processing to the image captured by the imaging element 2. The image processing unit 208 also acquires and generates various types of information from the captured image. The image processing unit 208 may be realized by a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. The image processing unit 208 may also be realized by a processor such as a DSP (Digital Signal Processor) that realizes a specific function by executing software.

[0033] Image processing by the image processing unit 208 includes pre-processing, color interpolation, correction, detection, data processing, evaluation value calculation, etc. Pre-processing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation is a process of interpolating values ​​of color components not included in image data, and is also called demosaic processing. Correction processing includes a process of adjusting white balance, a process of correcting image luminance, a process of correcting optical aberration of the photographing lens unit 1A, a process of correcting color, etc.

[0034] The detection process includes detection of characteristic regions (e.g., regions of faces, human bodies, animals, automobiles, etc.), tracking of characteristic regions, person recognition, etc. The image processing unit 208 detects information on the user's gaze position on the display surface of the display device 10 that displays images, and detects the subject in the image, thereby making it possible to identify the subject that the user is paying attention to. In addition, the image processing unit 208 can obtain motion characteristic information of the subject, such as how the subject in the image is moving on the image plane, by analyzing the difference between images along a time series.

[0035] The data processing includes scaling, encoding and decoding, header information generation, etc. The evaluation value calculation includes calculation of a pair of signals for AF processing using a phase difference signal, an evaluation value for contrast AF, an evaluation value used for automatic exposure control, etc. Note that the image processing performed by the image processing unit 208 is not limited to the above examples.

[0036] 4(a) and 4(b) are diagrams showing examples of the display of the field of view in the finder, showing a state in which the display device 10 is operating (a state in which an image is displayed). As shown in FIG. 4(a), the field of view in the finder includes a field of view mask 300, a focus detection area 400, and 180 ranging point indices 401. Each of the 180 ranging point indices 401 is displayed superimposed on a through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. The image processing unit 208 can obtain information on the user's gaze position on the display surface of the display device 10 from an eye image obtained by the eye imaging element 17 when the user looks through the finder.

[0037] In FIG. 4(a), the gaze position 421 is the position (area including the gaze position) where the gaze of the user is detected. The subject 422 is a person detected by the image analysis by the image processing unit 208. In the following description, when the detection target is a person, the hatched head area is detected as the subject. Since a movement accompanied by a fluctuation called fixation eye movement occurs in the human eyeball, it is preferable that the image processing unit 208 estimates a subject existing near the gaze position as a subject to which the gaze of the user is directed. When there is a deviation between the gaze position and the position where the subject exists, the image processing unit 208 displays an icon 423 indicating the subject to which the gaze of the user is directed. By displaying the icon 423, the user can confirm the subject that is estimated to be looked at by the user. Furthermore, the image processing unit 208 can estimate the subject that the user is looking at by directing the gaze of the user to the vicinity of the desired subject.

[0038] The vicinity of the gaze position can be a predetermined range including the gaze position. The predetermined range is determined based on, for example, the focal length, shooting distance, shooting mode information, and the size of the subject toward which the gaze is directed when capturing an image. The size of the subject is the size in the image, and can be expressed by, for example, the number of pixels occupied by the subject. The predetermined range may also be determined based on the type of subject, the moving speed, and the like. When multiple subjects are detected near the gaze position, the image processing unit 208 can estimate that the subject closest to the gaze position is the subject toward which the user's gaze is directed.

[0039] Fig. 4(b) shows an example in which multiple subjects exist in an image. In the image shown in Fig. 4(b), subjects 431 and 432 are detected. Gaze position 433 is the position where the user's gaze is detected. Icon 434 (corresponding to a first item) shows the subject (hereinafter also referred to as an estimated subject) that is estimated by image processing unit 208 to be directed at by the user's gaze.

[0040] When the user moves the line of sight to the vicinity of the subject 432, the image processing unit 208 changes the estimated subject from the subject 431 to the subject 432. The image processing unit 208 displays an icon 434 indicating the estimated subject for the subject 432, which is the destination of the line of sight. Note that the subject detected in the image is not limited to a person, and may be an animal, a vehicle, or the like.

[0041] <Explanation of gaze detection process> The gaze detection process (gaze detection method) for acquiring the gaze position will be described with reference to Figs. 5, 6(a), 6(b), and 7. The gaze of the right eye and the gaze of the left eye are detected by the following gaze detection method. Fig. 5 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for detecting the gaze. As shown in Fig. 5, the light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of the light receiving lens 16, and illuminate the user's eyeball 14. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is collected by the light receiving lens 16 on the eye imaging element 17. Fig. 6(a) is a schematic diagram of an eye image captured by the eye imaging element 17 (optical image of the eye projected on the eye imaging element 17), and Fig. 6(b) is a diagram showing the output intensity of the eye imaging element 17. Fig. 7 is a flowchart of the gaze detection process.

[0042] 7 starts, in step S1, CPU 3 drives light sources 13a and 13b using light source drive circuit 205 so as to emit infrared light toward user's eyeball 14. An optical image of the user's eye illuminated by infrared light passes through light receiving lens 16 and is formed on eye imaging element 17, which then performs photoelectric conversion. This provides an electrical signal of the eye image that can be processed.

[0043] In step S2, the CPU 3 acquires an eye image (image data, image signal) from the eye imaging element 17 via the line of sight detection circuit 201.

[0044] In step S3, the CPU 3 acquires the corneal reflection images Pd, Pe of the light sources 13a, 13b and the coordinates of a point corresponding to the center c of the pupil 141 from the eye image obtained in S2. The infrared light emitted from the light sources 13a, 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflection images Pd, Pe formed by a part of the infrared light reflected on the surface of the cornea 142 are collected by the light receiving lens 16 and imaged on the eye imaging element 17 to become the corneal reflection images Pd', Pe' in the eye image. Similarly, the light beams from the ends a, b of the pupil 141 are also imaged on the eye imaging element 17 to become the pupil end images a', b' in the eye image.

[0045] Figure 6(b) shows luminance information (luminance distribution) of region α in the eye image of Figure 6(a). In Figure 6(b), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. The coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe, and the coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb.

[0046] As shown in FIG. 6(b), at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe', an extremely high level of luminance is obtained. In the region from the coordinates Xa to the coordinates Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the ocular imaging element 17), an extremely low level of luminance is obtained except for the coordinates Xd and Xe. And in the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 143), an intermediate luminance between the above two types of luminance is obtained. For example, an intermediate luminance between the above two types of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is greater than the coordinate Xa and the region where the X coordinate is less than the coordinate Xb.

[0047] From the luminance distribution as shown in FIG. 6(b), the X coordinates Xd, Xe of the corneal reflection images Pd', Pe' and the X coordinates Xa, Xb of the pupil edge images a', b' can be obtained. For example, the coordinates of the corneal reflection images Pd', Pe' can be obtained as the coordinates of the corneal reflection images, and the coordinates of the pupil edge images a', b' can be obtained as the coordinates of the pupil edge images a', b'. In addition, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by forming an image of the light beam from the pupil center c on the ocular imaging element 17 can be expressed as Xc≒(Xa+Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this manner, the coordinates of the corneal reflection images Pd', Pe' and the coordinates of the pupil center image c' can be estimated.

[0048] In step S4, CPU 3 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of eyeball 14 relative to light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between corneal reflection images Pd', Pe'.

[0049] In step S5, CPU 3 calculates the rotation angle of the optical axis of eyeball 140 relative to the optical axis of light receiving lens 16. The X coordinate of the midpoint between corneal reflection images Pd and Pe almost coincides with the X coordinate of the center of curvature O of cornea 142. For this reason, if the standard distance from the center of curvature O of cornea 142 to the center of pupil c is Oc, the rotation angle θ of eyeball 14 in the ZX plane (plane perpendicular to the Y axis) X teeth It can be calculated by the following formula 1. The rotation angle θy of the eyeball 140 in the ZY plane (plane perpendicular to the X-axis) can also be calculated in a similar manner to the calculation of the rotation angle θx. β*Oc*SINθ X ≒{(Xd + Xe) / 2}-Xc (Eq. 1)

[0050] In step S6, the CPU 3 uses the rotation angles θx, θy calculated in step S5 to estimate the user's line of sight (hereinafter also referred to as the viewpoint) on the display device 10. If the coordinates (Hx, Hy) of the viewpoint correspond to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0051] Parameter m in Equations 2 and 3 is a constant determined by the configuration of the optical system for performing gaze detection processing, and is a conversion coefficient for converting rotation angles θx, θy into coordinates corresponding to the pupil center c on the display device 10. Parameter m is determined in advance and stored in the memory unit 4. Gaze correction parameters Ax, Bx, Ay, By are gaze correction parameters that correct individual differences in gaze. The gaze correction parameters Ax, Bx, Ay, By are acquired by performing calibration of gaze detection, and are stored in advance in the memory unit 4.

[0052] In step S7, CPU 3 stores the coordinates of the viewpoint and the time when the eye image was acquired (hereinafter, gaze detection time) in memory unit 4, and ends the gaze detection process.

[0053] The gaze detection method is not limited to the above method, and may be any method that detects the gaze based on an eye image. In addition, although an example in which the coordinates (Hx, Hy) of the viewpoint are obtained as the result (final result) of gaze detection has been described, any information related to the gaze (gaze information), such as rotation angles θx, θy, may be obtained as the result of gaze detection.

[0054] <Description of icon display control process> 8A to 8C, the display control of an icon indicating a subject at which a user's gaze is directed will be described. Fig. 8A is a flowchart illustrating an icon display control process in the first embodiment. The process shown in Fig. 8A includes image acquisition, subject detection, and icon display control processes. CPU 3 acquires images in chronological order, and repeatedly executes the process shown in Fig. 8A.

[0055] In step S8001, CPU 3 acquires an image captured by imaging element 2 and generated by image processor 208 through various image processing. In step S8002, CPU 3 detects a subject included in the image acquired in step S8001. Specifically, CPU 3 acquires information on the position of the head of a person detected from the image as subject position information. For example, in the image of FIG. 9(a), CPU 3 detects subject 901 and acquires information on the position of the head of subject 901.

[0056] In step S8003, CPU 3 acquires motion information (information on motion characteristics, information on actions) of the subject included in the image. CPU 3 can acquire motion information of the subject included in the image by comparing the image acquired in step S8001 with images previously generated by image processing unit 208. For example, FIGS. 9(a) to 9(c) are images generated at time t1, time t2 after time t1, and time t3 after time t2, respectively. In this case, CPU 3 can acquire motion information that subject 901 is moving at a substantially constant speed. Also, as shown in FIG. 9(d), multiple subjects including subject 901 are detected in the image. If all of the objects exhibit similar motion characteristics, then the objects are treated as an object group 902.

[0057] In step S8004, CPU 3 performs the gaze detection process described in Figs. 5 to 7 using eye imaging element 17 and gaze detection circuit 201 to obtain gaze position information. In step S8005, CPU 3 determines whether gaze detection (obtaining gaze position information) in step S8004 was successful. If gaze detection was successful, the process proceeds to step S8007. If gaze detection was unsuccessful, the process proceeds to step S8006.

[0058] In step S8006, CPU 3 maintains the display position of the icon (first item) in the previous process (processing for the image captured before the current image) and the display state of whether to display it, and returns to step S8001. Specifically, if the icon was not displayed in the previous process, CPU 3 controls not to display the icon for the current image either, and if the icon was displayed in the previous process, CPU 3 controls to display the icon for the current image in the same position as the previous image.

[0059] If the gaze detection fails, the CPU 3 may display the icon in a different manner from when the gaze detection succeeds in order to inform the user that the gaze detection has failed. For example, if the gaze detection fails, the CPU 3 may display the icon in a different color from when the gaze detection succeeds, or may display the icon in a blinking manner. Note that, in order to prevent the visibility from decreasing due to frequent changes in the color of the icon, the CPU 3 may change the display manner of the icon after a predetermined time has elapsed since it was determined that the gaze detection has failed.

[0060] In step S8007, CPU 3 determines whether or not a subject is detected at the gaze position acquired in step S8004. When a subject detected in step S8002 exists near the user's gaze position, CPU 3 can determine that a subject is detected at the gaze position. If a subject is detected at the gaze position, processing proceeds to step S8009. If a subject is not detected at the gaze position, processing proceeds to step S8008.

[0061] In step S8008, CPU 3 displays an icon at the gaze position acquired in step S8004. For example, in Fig. 10, the user's gaze position is located away from subject 1001 included in the image, so CPU 3 displays icon 1002 (second item) indicating the detected user's gaze position. For example, when subject 1001 included in the image is away from the user's gaze position by a predetermined distance or more, CPU 3 displays an icon indicating the user's gaze position.

[0062] When displaying icon 1002 indicating the gaze position, since there is no subject at the user's gaze position, CPU 3 does not display an icon (first item) indicating the subject at which the user's gaze is directed. This allows CPU 3 to inform the user that subject 1001 is not captured as the subject at which the user's gaze is directed.

[0063] Area 1003 is an area indicating the vicinity of the gaze position of the user. CPU 3 sets area 1003 as an area for determining whether or not a subject is present in the vicinity of the gaze position. Area 1003 may be set by the user using operation members 41 to 43. CPU 3 may also set area 1003 automatically according to the optical state of the lens, such as the focal length of photographing lens unit 1A or the photographing object distance at the in-focus position, or may set area 1003 automatically according to the mode setting of the AF function set in camera 1. In other words, the vicinity of the gaze position is an area that includes the gaze position of the user and can be set by the user or CPU 3.

[0064] In step S8009, CPU 3 determines whether to display an icon indicating a subject at which the user's line of sight is directed, based on the status of the user's image capture with camera 1. The status of the image capture with camera 1 includes, for example, statuses such as whether the user is paying attention to the subject estimated by image processing unit 208, whether the imaging range of the image has changed, and whether the attitude of the imaging device capturing the image has changed.

[0065] The process of step S8009 will be described in detail with reference to Fig. 8B. In step S8201, CPU 3 determines whether or not the user, who is the photographer, is paying attention to a subject that he or she wishes to photograph (a subject that is estimated to be directed at the user's gaze, an estimated subject).

[0066] Whether the user is paying attention to the estimated subject is determined from the eye image captured by the eye image sensor 17 based on the degree of stability of the user's gaze position, the presence or absence or frequency of the user's blinking, etc. For example, the CPU 3 can determine that the user is paying attention to the estimated subject when the amount of movement of the user's gaze position in a predetermined time is smaller than a first threshold value, or when the number of blinks in a predetermined time is smaller than a second threshold value. Since the degree of stability of the gaze position and the frequency of blinking vary from person to person, the first threshold value and the second threshold value may be set in advance for each user. If the user is paying attention to the estimated subject, the process proceeds to step S8202. If the user is not paying attention to the estimated subject, the process proceeds to step S8203.

[0067] In step S8202, CPU3 determines to display an icon indicating the estimated subject. By displaying the icon indicating the estimated subject, CPU3 can inform the user that the subject the user is paying attention to is captured as a subject to be captured. For example, when the user is waiting for the subject he / she wants to photograph at a dolphin show to move to a desired position, or when the user is waiting for a photo opportunity in front of a soccer goal, the icon indicating the subject the user is paying attention to is displayed, so that the user does not miss the photo opportunity.

[0068] In step S8203, CPU 3 determines whether or not a zoom operation, pan operation, or tilt operation has been performed. Zoom operation includes a user's zoom operation on photographing lens unit 1A and a digital zoom operation on camera 1. CPU 3 determines whether or not a zoom operation, pan operation, or tilt operation has been performed, thereby determining whether or not the imaging range of the image has changed.

[0069] For example, the CPU 3 can determine that the imaging range of the image has changed when a zoom operation is performed, when the amount of change in focal length due to the zoom operation is greater than a third threshold, or when the amount of change in zoom magnification due to the zoom operation is greater than a fourth threshold. The CPU 3 can also determine that the imaging range of the image has changed when a panning operation or a tilting operation is performed, or when the amount of movement of the imaging range due to the panning operation or the tilting operation is greater than a fifth threshold. The third to fifth thresholds can be determined based on, for example, whether or not there is a possibility that the subject on which the user is directing his or her gaze will be lost due to the change in the imaging range. If a zoom operation, a panning operation, or a tilting operation is performed, the process proceeds to step S8202. If a zoom operation, a panning operation, or a tilting operation is not performed, the process proceeds to step S8204.

[0070] In step S8202, CPU 3 determines to "display an icon" indicating the estimated subject. When the composition is changed by zooming or the like, the position of the subject that the user was directing his or her gaze at may change. If a different subject comes to be in the position of the subject that the user was originally directing his or her gaze at, there is a possibility that the user may lose sight of the subject that the user was attempting to photograph. Therefore, CPU 3 displays an icon indicating the estimated subject when the imaging range of the image has changed by zooming or the like, thereby enabling the user to easily find the subject that the user was attempting to photograph. You can make sure you don't lose it.

[0071] 11(a) and 11(b) are diagrams for explaining icon display based on a zoom operation. Fig. 11(a) shows a state in which a subject 1101 is estimated as a subject at which the user's gaze is directed when zoomed in to the TELE side. Since there is only one subject near gaze position 1102 (the icon indicating gaze position 1102 is not displayed on the display device 10), the icon indicating the subject at which the user's gaze is directed is not displayed.

[0072] Fig. 11(b) shows the situation after zoom operation in the WIDE direction from the state of Fig. 11(a). In Fig. 11(b), a subject 1103 different from subject 1101 is detected in the image. If the gaze position 1102 is the same as before the zoom operation, the user may be confused as to whether subject 1103 has been captured as an estimated subject. Therefore, when the user is directing his / her gaze at subject 1101, CPU 3 determines to display an icon for subject 1101 in order to inform the user that subject 1101 has been captured as an estimated subject.

[0073] In step S8204, CPU 3 determines whether the attitude of camera 1 has changed. When shooting with camera 1, the user alternates between a horizontal position where the image is shot so that the long side of the image is horizontal, and a vertical position where the image is shot so that the long side of the image is vertical. CPU 3 determines whether the attitude of camera 1 has changed, for example, by changing the camera from a horizontal position to a vertical position. CPU 3 can obtain information on the attitude of camera 1 using attitude detection unit 207.

[0074] CPU 3 may determine whether the attitude of camera 1 has changed after acquiring information on the user's gaze position. If the attitude of camera 1 is changed while looking through the viewfinder, an icon indicating the estimated subject is displayed, making it easier for the user to confirm the desired subject. If the attitude of camera 1 has changed, processing proceeds to step S8202. If the attitude of camera 1 has not changed, processing proceeds to step S8205.

[0075] In step S8202, CPU 3 determines to display an icon indicating the estimated subject. This is because when the attitude of camera 1 changes, the user's line of sight is likely to become unstable, and this is to prevent the user from losing sight of the subject that was being viewed before the attitude of camera 1 changed.

[0076] 12(a) and 12(b) are diagrams for explaining icon display based on a change in the posture of the camera 1. FIG. 12(a) shows an example of an image captured in a horizontal position. In FIG. 12(a), subjects 1201 and 1202 are detected. The user's gaze is directed toward the subject 1201. FIG. 12(b) shows an example of an image after the user changes the position of the camera 1 from the state shown in FIG. 12(a) to a vertical position. While the camera 1 is changed from the horizontal position to the vertical position, the position of the user's gaze becomes unstable due to the change in composition. To prevent the user from mistakingly recognizing the subject 1202 as the subject 1201 that he had been looking at, the CPU 3 determines to display an icon 1203 indicating the estimated subject.

[0077] In step S8205, CPU 3 determines that it is undecided whether to display an icon indicating an estimated subject ("icon display undecided"). In other words, if none of the conditions in steps S8201, S8203, and S8204 is satisfied, CPU 3 determines whether to display an icon indicating an estimated subject based on the state of the subject in step S8020 in FIG. 8A, which will be described later.

[0078] In step S8206, the CPU 3 determines whether an icon indicating the estimated subject is currently displayed. If the icon indicating the estimated subject is in a hidden state, the process proceeds to step S8207. If the icon indicating the estimated subject is in a displayed state, the process shown in FIG. 8B ends.

[0079] In step S8207, CPU 3 records the current time as a reference time in memory unit 4. Step S8207 is a process performed in the case where the icon indicating the estimated subject is currently in a non-display state and it is determined in step S8202 that the icon is to be displayed. By recording the time (reference time) at which it is determined that the icon will transition from a non-display state to a display state, CPU 3 can measure the time since the icon transitioned to a display state.

[0080] As described above, by the process shown in Fig. 8B, CPU 3 determines whether or not to display an icon of a subject at which the user's gaze is directed, based on the imaging situation (the situation of the photographer). CPU 3 ends the process of Fig. 8B, and proceeds to step S8010 of Fig. 8A.

[0081] In step S8010 of Fig. 8A, CPU 3 determines whether or not the result of the display determination of the icon indicating the estimated subject in step S8009 was "display icon". If the determination result is "display icon", the process proceeds to step S8011. If the determination result is "icon display is undecided", the process proceeds to step S8020.

[0082] In step S8011, CPU 3 determines whether or not a predetermined time has elapsed since the reference time recorded in step S8207 in Fig. 8B. That is, CPU 3 determines whether or not a predetermined time has elapsed since the time (reference time) when the icon representing the estimated subject transitioned from a non-display state to a display state. If the predetermined time has elapsed since the reference time, the process proceeds to step S8013. If the predetermined time has not elapsed since the reference time, the process proceeds to step S8012.

[0083] In step S8012, CPU 3 displays an icon indicating the estimated subject (for example, icon 423 in FIG. 4(a)). In step S8013, CPU 3 hides the icon indicating the estimated subject. After a predetermined time has elapsed since the imaging situation changed and an icon indicating the estimated subject was displayed, it is expected that the user will be able to stably direct their gaze toward a desired subject, so CPU 3 hides the icon indicating the estimated subject.

[0084] Although an example has been shown in which the icon indicating the estimated subject is automatically hidden when a predetermined time has elapsed since the icon indicating the estimated subject was switched from a non-display state to a display state, the CPU 3 may maintain the display state of the icon. Also, the predetermined time from the reference time until the icon indicating the estimated subject is hidden may be set by the user. In steps S8012 and S8013, when the icon indicating the estimated subject is respectively switched to a display state and a non-display state, the process returns to step S8001.

[0085] In step 8020, CPU 3 determines whether to display an icon indicating the subject at which the user's gaze is directed, based on the subject's status. The subject's status includes, for example, whether the number of subjects included in the image has changed, whether the number of subjects included in the image is two or more, whether the estimated subject is accelerating or decelerating, and whether the estimated subject is in focus.

[0086] The process of step S8020 will be described in detail with reference to FIG. 8C. In step S8301, CPU 3 determines whether or not there has been an increase or decrease in the number of subjects in an image since the previous image (frame). The number of subjects is not limited to the number of subjects in an image, and may be the number of subjects detected in the vicinity of the gaze position. Specifically, the number of subjects may be the number of subjects detected in a predetermined range including the estimated subject at which the gaze is directed. The predetermined range may be, for example, For example, the detection target can be determined based on at least one of the focal length, shooting distance, shooting mode information, and the size of the estimated subject when capturing the image. The subject to be detected is a person, an animal, a car, or the like that can be detected by analyzing the image.

[0087] The CPU 3 may detect a predetermined type of subject from among the subjects included in the image. For example, if the user sets animals as the predetermined type, the CPU 3 detects animals as the subject. The user may set multiple types of subjects as the predetermined type. If there is an increase or decrease in the number of detected subjects, the process proceeds to S8302. If there is no increase or decrease in the number of detected subjects, the process proceeds to S8303.

[0088] In step S8302, CPU 3 determines to display an icon indicating the estimated subject. By displaying an icon indicating the estimated subject, even if the number of subjects increases or decreases, the user can easily confirm the subject that the user is looking at.

[0089] In step S8303, CPU 3 determines whether or not the number of subjects detected near the gaze position is 1. If the number of subjects is 1, the process proceeds to step S8304. If the number of subjects is more than one, the process proceeds to step S8307.

[0090] In step S8304, CPU 3 determines whether or not one object existing in the vicinity of the gaze position in the screen is in focus. If the object is in focus, the process proceeds to step S8305. If the object is not in focus, the process proceeds to step S8306.

[0091] In step S8305, CPU 3 determines to "hide the icon." In step S8306, CPU 3 determines to "display the icon." CPU 3 can notify the user that the subject at which the user is looking is not in focus by displaying the icon when the subject is out of focus.

[0092] 13(a) and 13(b) are diagrams illustrating icon display based on the change in the number of subjects described in the processing of steps S8301 to S8306. Fig. 13(a) shows a situation where, at time t11, subject 1301 is present near gaze position 1302. Since it is clear that the subject 1301 is the object toward which the user's gaze is directed, the icon indicating the estimated subject and the icon indicating gaze position 1302 are not displayed.

[0093] Fig. 13(b) shows a state in which a different subject 1303 has entered the composition after the situation in Fig. 13(a). Because subject 1303 has appeared near gaze position 1304, it is preferable for CPU 3 to notify the user of whether the gaze was directed toward subject 1301 or subject 1303. Therefore, CPU 3 displays icon 1305 indicating the subject toward which the user's gaze was directed. Note that the icon indicating gaze position 1304 is not displayed.

[0094] In step S8307, CPU 3 determines whether or not a subject present near the user's gaze position is stationary or moving at a constant speed. Step S8307 is a process executed when multiple subjects are present near the gaze position. CPU 3 acquires information on the movement characteristics of the subjects near the gaze position (movement information). The movement information can be acquired in the same manner as in step S8003 of FIG. 8A described using FIGS. 9(a) to 9(c). Depending on the movement characteristics (movement) of the subjects, it is preferable to display an icon indicating the subject at which the user's gaze is directed so that the user does not lose sight of the desired subject.

[0095] When the CPU 3 determines based on the motion information that the subject in the vicinity of the gaze position is stationary or moving at a constant speed, the process proceeds to step S8308. If it is determined that any of the subjects present in the vicinity is accelerating or decelerating while moving, the process proceeds to step S8310.

[0096] In step S8308, CPU 3 determines whether the subject the user is looking at is in focus. If the subject the user is looking at is in focus, the process proceeds to step S8309. If the subject the user is looking at is out of focus, the process proceeds to step S8310.

[0097] In step S8309, CPU 3 determines to “hide the icon.” When the subject is in focus, it is assumed that the user's line of sight is stably directed toward the desired subject, and therefore it is determined to hide the icon indicating the estimated subject.

[0098] In step S8310, CPU 3 determines to "display an icon." If the subject is out of focus, it is assumed that the user is not stable, and so it is determined to display an icon indicating the presumed subject. Also, if a subject present in the vicinity of the gaze position is moving while accelerating or decelerating (for example, when performing vigorous movements such as dancing), CPU 3 determines to display an icon indicating the subject at which the user is directing his or her gaze. By displaying an icon indicating the subject at which the user is directing his or her gaze, the user is less likely to lose sight of the desired subject.

[0099] 14(a) to 14(d) are diagrams for explaining icon display based on subject movement information described in the processing of steps S8307 to S8310. Fig. 14(a) is an example of so-called portrait photography, in which multiple subjects are stationary. Subject 1401 detected from the image is assumed to be out of focus. CPU 3 estimates that the user's gaze is directed at subject 1401 present in the vicinity of gaze position 1402, and displays icon 1403 indicating that subject 1401 is an estimated subject.

[0100] 14(b) shows a situation in which subject 1401 has entered an in-focus state. CPU 3 displays icon 1404 indicating that subject 1401 is in-focus, and does not display icon 1403 indicating that subject 1401 is an estimated subject. By preferentially displaying icon 1404 indicating the in-focus state, CPU 3 can reduce the amount of information to be displayed and ensure visibility. CPU 3 can also notify the user in a timely manner that subject 1401 has entered an in-focus state.

[0101] Fig. 14(c) shows a situation where a subject group 1414 including a subject 1411 near the gaze position and a plurality of surrounding subjects have similar movement characteristics. Fig. 14(c) shows the situation at time t21. CPU 3 displays icon 1413 indicating that subject 1411 near gaze position 1412 is an estimated subject.

[0102] Fig. 14(d) shows a situation after the situation in Fig. 14(c), where a group of subjects 1414 including subject 1411 and multiple surrounding subjects are moving in the same direction at a substantially constant speed. Fig. 14(d) shows the situation at time t22. At this time, a subject 1415 different from subject 1411 is present near gaze position 1412. Therefore, the subject at which the user's gaze is directed is estimated to be subject 1415.

[0103] In FIG. 14(d), it is assumed that subject 1415 is in focus. At time t22, subject 1411 and subject 1415 included in subject group 1414 are moving with similar movement characteristics. CPU 3 displays icon 1416 indicating that subject 1415 is in focus, but does not display an icon indicating that subject 1415 is an estimated subject. CPU 3 does not display an icon indicating that subject 1415 is an estimated subject, but displays icon 141 By giving priority to displaying 6, it is possible to suppress the amount of information to be displayed and ensure visibility. Also, the CPU 3 can notify the user in a timely manner that the subject 1415 has come into focus.

[0104] In step S8312, CPU 3 determines whether it has been determined in the processing of steps S8301 to S8310 that "icons are to be displayed" or that "icons are to be hidden." If it has been determined that "icons are to be displayed," the processing proceeds to step S8313. If it has been determined that "icons are to be hidden," the processing shown in FIG. 8C ends.

[0105] In step S8313, CPU 3 determines whether the icon indicating the estimated subject is currently in a display state or a non-display state. If the icon indicating the estimated subject is in a non-display state, the process proceeds to step S8314. If the icon indicating the estimated subject is in a display state, the process shown in FIG. 8C ends.

[0106] In step S8314, CPU 3 records the current time as a reference time in memory unit 4. Step S8314 is a process performed when the icon indicating the estimated subject is currently in a non-display state and it is determined that the icon is to be "displayed." By recording the time (reference time) at which it is determined that the icon will be switched from a non-display state to a display state, CPU 3 can measure the time since the icon transitioned to a display state.

[0107] As described above, by the process shown in Fig. 8C, CPU 3 determines whether or not to display an icon of a subject at which the user's gaze is directed, based on the state of the subject. CPU 3 ends the process in Fig. 8C, and proceeds to step S8021 in Fig. 8A.

[0108] In step S8021 of Fig. 8A, CPU 3 determines whether or not the result of the display determination of the icon indicating the estimated subject in step S8020 was "display the icon". If the determination result is "display the icon", the process proceeds to step S8022. If the determination result is "hide the icon", the process proceeds to step S8025.

[0109] In step S8022, CPU 3 determines whether or not a predetermined time has elapsed since the reference time recorded in step S8314 of Fig. 8C. That is, CPU 3 determines whether or not a predetermined time has elapsed since the time (reference time) when the icon representing the estimated subject transitioned from a non-display state to a display state. If the predetermined time has elapsed since the reference time, the process proceeds to step S8024. If the predetermined time has not elapsed since the reference time, the process proceeds to step S8023.

[0110] In step S8023, CPU 3 displays an icon indicating the subject (estimated subject) at which the user's gaze is directed. In step S8024, CPU 3 hides the icon indicating the estimated subject. When the icon indicating the estimated subject becomes a displayed state and a hidden state in steps S8023 and S8024, respectively, the process returns to step S8001.

[0111] In step S8025, CPU 3 hides the icon indicating the subject (estimated subject) at which the user's gaze is directed, in response to the determination result in step S8020. In step S8026, CPU 3 initializes the reference time recorded in step S8207 or step S8314, and returns to step S8001.

[0112] According to the above-mentioned embodiment 1, the camera 1 controls whether to display an icon indicating a subject (estimated subject) that is estimated to be the subject at which the user is directing his / her gaze, based on at least one of the situation of capturing the image and the situation of the subject included in the image. The camera 1 automatically displays or hides the icon indicating the estimated subject, thereby improving visibility according to the situation. It is therefore possible to achieve both high reliability and improved operability.

[0113] The icon display control process described in Fig. 8A shows an example of a process in which icon display determination based on the imaging situation (Fig. 8B) is followed by icon display determination based on the subject situation (Fig. 8C), but the order of determination is not limited to this. The CPU 3 may determine whether or not to display an icon indicating an estimated subject by determining at least one of the various conditions described as the imaging situation and the various conditions described as the subject situation in any order.

[0114] [Embodiment 2] In the first embodiment, the camera 1 determines the focus state of a subject existing near the gaze position, and when the subject is in focus, controls so as to hide an icon indicating the subject (estimated subject) at which the user's gaze is directed. On the other hand, as a general shooting method, there are cases where the composition is changed after focusing on a specific subject (focus lock), and cases where the gaze position is changed to search for other subjects while maintaining the focus position. The second embodiment is an embodiment in which an icon indicating an estimated subject is displayed when the user changes the composition or gaze position after the subject is in focus. By continuing to display the icon indicating the estimated subject even after the subject is in focus, operability regarding focus lock and searching for other subjects is improved.

[0115] Fig. 15 is a flowchart of the icon display determination process according to embodiment 2. The process in Fig. 15 is a process in which steps S8501 to S8504 and steps S8505 to S8513 are added to the icon display determination process based on the subject situation in Fig. 8C. The same processes as in Fig. 8C are denoted by the same reference numerals and detailed description thereof will be omitted.

[0116] If it is determined in step S8304 that one subject existing near the gaze position on the screen is in focus, the process proceeds to step S8501. In step S8501, one subject exists near the gaze position on the screen, and the subject is in focus. In step S8501, CPU 3 determines whether or not the focus position is fixed for the in-focus subject in a focus lock state, and the gaze position is in a position different from the subject to be focus locked (in-focus subject).

[0117] Whether or not the gaze position is at a position different from the subject to which focus lock is applied can be determined, for example, by whether or not the distance between the subject to which focus lock is applied (subject in focus) and the gaze position is equal to or greater than a predetermined threshold. When the distance between the subject to which focus lock is applied and the gaze position is equal to or greater than a predetermined threshold, the CPU 3 determines that the gaze position is at a position different from the subject to which focus lock is applied. The predetermined threshold may be dynamically switched based on the shooting mode of the camera 1, the state of the lens such as the focal length, and the like.

[0118] If the gaze position is not at the position of the subject to focus lock, the process proceeds to step S8502. If the gaze position is not at the position of the subject to focus lock, the process proceeds to step S8504.

[0119] In step S8502, CPU 3 determines whether or not a subject other than the subject targeted for focus lock is present near the gaze position. If a subject is present near the gaze position, the process proceeds to step S8503. If a subject is not present near the gaze position, the process proceeds to step S8504.

[0120] In step S8503, the CPU 3 determines to "display the icon." In step S8504, the CPU 3 determines to "hide the icon." When the user is looking at a subject other than the subject targeted for focus lock, the CPU 3 can notify the user of the presumed subject by displaying an icon indicating the subject at which the user's gaze is directed.

[0121] If it is determined in step S8308 that the subject at which the user is directing his or her gaze is in focus, the process proceeds to step S8505. In step S8505, multiple subjects are present in the vicinity of the gaze position. The multiple subjects are stationary or moving at a constant speed and in focus. In step S8505, CPU 3 determines whether the in-focus subject is stationary or moving at a constant speed. If the in-focus subject is stationary, the process proceeds to step S8506. If the in-focus subject is moving at a constant speed, the process proceeds to step S8510.

[0122] In step S8506, in a focus lock state in which the focus position is fixed with respect to a focused subject, CPU 3 determines whether or not the gaze position is at a position different from the focus lock target subject (focused target subject). If the gaze position is at a position different from the focus lock target subject, the process proceeds to step S8507. If the gaze position is at the position of the focus lock target subject, the process proceeds to step S8509.

[0123] In step S8507, CPU 3 determines whether or not a subject other than the subject targeted for focus lock is present near the gaze position. If a subject is present near the gaze position, the process proceeds to step S8508. If a subject is not present near the gaze position, the process proceeds to step S8509.

[0124] In step S8508, CPU 3 determines to "display icon." In step S8509, CPU 3 determines to "hide icon." When the user is looking at a subject other than the subject targeted for focus lock, CPU 3 can notify the user of the subject at which the user's gaze is directed by displaying an icon indicating the presumed subject.

[0125] 16(a) to 16(d) are diagrams for explaining icon display in the focus lock state. The subject 1401, gaze position 1402, icon 1403, and icon 1404 in FIG. 16(a) and FIG. 16(b) are the same as those in FIG. 14(a) and 14(b) of the first embodiment. In FIG. 16(a), the subject 1401 is present near the gaze position 1402. In FIG. 16(b), the subject 1401 is in focus, and icon 1404 indicating the in-focus state is displayed. Meanwhile, icon 1403 indicating the estimated subject is not displayed. In addition, CPU 3 sets the focus to the subject 1401 in a locked state.

[0126] Fig. 16(c) shows a situation in which the user's gaze has moved from the state in which focus was locked on subject 1401 in Fig. 16(b) to gaze position 1601 that is different from subject 1401. Even if the user's gaze has moved to gaze position 1601, if the focus is locked on subject 1401, CPU3 maintains the display of icon 1404 indicating that subject 1401 is in focus. Since there is no subject near gaze position 1601, CPU3 does not display the icon indicating the subject at which the user's gaze is directed (first item) and the icon indicating the gaze position (second item).

[0127] FIG. 16(d) shows a situation in which the user's gaze has moved from the situation in FIG. 16(c) to gaze position 1602. Even if the user's gaze moves to gaze position 1601, the focus is locked on subject 1401, so CPU 3 maintains the display of icon 1404 indicating that subject 1401 is in focus. CPU 3 also changes the display of icon 1404, which indicates that subject 1401 is in focus, to subject 1603 present in the vicinity of gaze position 1602, and displays the icon indicating the subject on which the user's gaze is directed. By displaying the icon 1404 indicating the in-focus position and the icon 1604 indicating the estimated subject, the CPU 3 can notify the user of another desired subject to which the user has directed his or her line of sight, in addition to the in-focus position.

[0128] Although the focus lock function generally operates when the autofocus mode is the one-shot mode, the second embodiment can also be applied to other modes such as the servo mode. For example, when the subject 1401 is in focus in the servo mode and the user moves his / her gaze to another subject, the CPU 3 may control the display of each icon in the same manner as the processing described in Figs. 16(c) and 16(d).

[0129] In step S8510, the in-focus subject is moving at a constant speed, and CPU 3 determines whether the gaze position is at a position different from that of the in-focus subject (a state in which focus tracking of a moving subject is successful). If the gaze position is at a position different from that of the subject to be focus-locked, the process proceeds to step S8511. If the gaze position is at the position of the in-focus subject, the process proceeds to step S8513.

[0130] In step S8511, CPU 3 determines whether or not a subject other than the subject for which focus tracking has been successful exists near the gaze position. If a subject exists near the gaze position, the process proceeds to step S8512. If a subject does not exist near the gaze position, the process proceeds to step S8513.

[0131] In step S8512, CPU 3 determines to "display the icon." In step S8513, CPU 3 determines to "hide the icon." When the user is looking at a subject other than the subject undergoing constant-speed movement for which focus tracking has been successful, CPU 3 can notify the user of the subject at which the user's gaze is directed by displaying an icon indicating the presumed subject.

[0132] 17(a) to 17(c) are diagrams for explaining icon display in the focus tracking state. The subject 1411, gaze position 1412, icon 1413, subject 1415, and icon 1416 in Fig. 17(a) and Fig. 17(b) are the same as Fig. 14(c) and 14(d) in the first embodiment. The subject 1411 and the subject 1415 are moving at a constant speed (operating at a constant speed), and in Fig. 17(b), the subject 1415 existing near the gaze position 1412 is in focus. The CPU 3 displays the icon 1416 indicating that the subject 1415 is in focus, and does not display the icon indicating the subject (subject 1415 in Fig. 17(b)) at which the user's gaze is directed.

[0133] In Fig. 17(c), the CPU 3 performs a tracking or predicting operation of the focus position in a servo AF mode (a mode that keeps the focus adjusted according to the movement of the subject) for the subject 1415. Fig. 17(c) shows a situation in which the subject 1415 is in focus and focus tracking has been successful, and the user's gaze has moved to gaze position 1701.

[0134] Even if the user's gaze moves to gaze position 1701, if focus tracking of subject 1415 has been successful, CPU 3 maintains the display of icon 1416 indicating that subject 1415 is in focus. CPU 3 also displays icon 1703 indicating the subject at which the user's gaze is directed, for subject 1702 that exists in the vicinity of gaze position 1701. By displaying icon 1416 indicating the in-focus position and icon 1703 indicating the estimated subject, CPU 3 can notify the user of other desired subjects at which the user has directed his or her gaze, in addition to the in-focus position.

[0135] According to the second embodiment, the camera 1 focuses on the subject and estimates the subject. Even if the icon indicating that the subject is a subject is hidden, when the user's gaze shifts to another subject, an icon indicating that the other subject is an estimated subject is displayed. That is, in a state where camera 1 is in focus or in focus tracking on a subject near the gaze position, camera 1 hides the icon indicating the subject on which the gaze is directed in order to improve visibility. Then, when the user shifts his / her gaze to another subject, camera 1 displays an icon indicating the subject on which the user's gaze is directed (estimated subject) for the subject near the gaze position. By displaying an icon indicating the estimated subject when the user directs his / her gaze to another subject while in focus or in focus tracking on the subject, camera 1 can ensure both visibility and improve operability.

[0136] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0137] For example, although an example of viewing a subject through an EVF (electronic viewfinder) has been described, the present invention is also applicable to viewing a subject through an OVF (optical viewfinder). The present invention is applicable to devices other than imaging devices (cameras), and is also applicable to viewing three-dimensional computer graphics, etc. The present invention is applicable to various electronic devices capable of detecting the viewpoint.

[0138] [Applications to other electronic devices] Fig. 18(a) is an external view of a notebook type personal computer 1810 (notebook PC). In Fig. 18(a), an imaging unit 1815 that captures an image of a user looking at a display unit 1811 of the notebook PC 1810 is connected to the notebook PC 1810, and the notebook PC 1810 obtains the imaging result from the imaging unit 1815. Then, the notebook PC 1810 detects the user's viewpoint based on the imaging result. The present invention is also applicable to the notebook PC 1810.

[0139] Fig. 18(b) is an external view of a smartphone 1820. In Fig. 18(b), the smartphone 1820 detects the viewpoint of a user looking at a display unit 1822 of the smartphone 1820 based on an imaging result of an in-camera 1821 (front camera). The present invention is also applicable to the smartphone 1820. Similarly, the present invention is also applicable to various tablet terminals.

[0140] FIG. 18(c) is an external view of the game machine 1830. In FIG. 18(c), a head mounted display 1835 (HMD) that displays a VR (Virtual Reality) image of a game on a display unit 1836 is connected to the game machine 1830. The HMD 1835 has a camera 1837 that captures an image of the eye of a user wearing the HMD 1835, and the game machine 1830 acquires the image capture result from the HMD 1835. Then, the game machine 1830 detects the user's viewpoint based on the image capture result. The present invention is also applicable to the game machine 1830. The components of the game machine 1830 may be built into the HMD 1835. Therefore, the present invention is also applicable to the HMD 1835. Just as the present invention is applicable to a case where a VR image displayed on an HMD is viewed, the present invention is also applicable to a case where an AR (Augmented Reality) image displayed on the lens part of a glasses-type wearable terminal or an actual object beyond the lens part is viewed. Just as the present invention is applicable to VR technology and AR technology, the present invention is also applicable to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

[0141] [Other embodiments] The present invention relates to a program for implementing one or more functions of the above-described embodiments, which is transmitted via a network or The program may be provided to a system or device via a storage medium, and may be read and executed by one or more processors in the computer of the system or device. The program may also be implemented by a circuit (e.g., ASIC) that implements one or more functions.

[0142] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) An acquisition means for acquiring information on a gaze position of a user viewing a captured image; an estimation means for estimating a subject at which the user's gaze is directed based on the user's gaze position; a control means for controlling whether or not to display a first item indicating the subject estimated by the estimation means, based on at least one of a capture situation of the image and a situation of the subject included in the image; 1. An electronic device comprising: (Configuration 2) The control means performs control so as to display a second item indicating the user's gaze position when the user's gaze position is located away from a subject included in the image. 2. The electronic device according to configuration 1. (Configuration 3) The control means When the user is paying attention to the subject estimated by the estimation means, When the imaging range of the image changes, or When the attitude of the imaging device that captures the image changes, Control to display the first item 3. The electronic device according to configuration 1 or 2. (Configuration 4) The control means determines that the user is paying attention to the subject estimated by the estimation means when an amount of movement of the user's gaze position in a predetermined time is smaller than a first threshold value, or when a number of blinks in a predetermined time is smaller than a second threshold value. 4. The electronic device according to configuration 3. (Configuration 5) The control means determines that the imaging range of the image has changed when a zoom operation is performed, when a change in focal length due to the zoom operation is greater than a third threshold value, or when a change in zoom magnification due to the zoom operation is greater than a fourth threshold value. 4. The electronic device according to configuration 3. (Configuration 6) The control means determines that the imaging range of the image has changed when a panning operation or a tilting operation is performed, or when a movement amount of the imaging range due to the panning operation or the tilting operation is greater than a fifth threshold value. 4. The electronic device according to configuration 3. (Configuration 7) The control means determines whether or not the attitude of the imaging device has changed in a state in which information on the user's gaze position has been acquired. 4. The electronic device according to configuration 3. (Configuration 8) The control means When the number of objects included in the image changes, If the number of subjects included in the image is two or more, When the subject estimated by the estimation means is accelerating or decelerating, or When the subject estimated by the estimation means is not in focus, Control to display the first item 8. The electronic device according to any one of configurations 1 to 7. (Configuration 9) The number of subjects is the number of subjects of a predetermined type among the subjects included in the image. 9. The electronic device according to configuration 8. (Configuration 10) The number of subjects is the number of subjects detected within a predetermined range including the subject estimated by the estimation means. 10. The electronic device according to configuration 8 or 9. (Configuration 11) The predetermined range is determined based on at least one of the focal length, shooting distance, shooting mode information when capturing the image, and the size of the subject estimated by the estimation means. 11. The electronic device according to configuration 10. (Configuration 12) The control means performs control so as to display the first item when the subject estimated by the estimation means is in a focused state and a gaze position of the user is detected with respect to a subject different from the focused subject. 12. The electronic device according to any one of configurations 8 to 11. (Configuration 13) When the control means fails to acquire information on the user's gaze position, If the first item was not displayed for an image captured before the image, the first item is not displayed for the image; If the first item is displayed in an image captured before the image, the first item is displayed in the image at the same position as in the image captured before the image. Control so that 13. The electronic device according to any one of configurations 1 to 12. (Configuration 14) When the control means fails to acquire information on the user's gaze position and displays the first item on the image, the control means displays the first item in a manner different from a manner in which the control means successfully acquires information on the user's gaze position. 14. The electronic device according to configuration 13. (method) An acquisition step of acquiring information on a gaze position of a user viewing a captured image; an estimation step of estimating a subject at which the user's gaze is directed based on the user's gaze position; a control step of controlling whether or not to display a first item indicating the subject estimated in the estimation step, based on at least one of a capture situation of the image and a situation of the subject included in the image; 13. A method for controlling an electronic device comprising: (program) A program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 14. (medium) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 14. [Explanation of symbols]

[0143] 1: camera (electronic device), 3: CPU, 10: display device, 11: display device drive circuit, 17: eye image sensor, 201: line of sight detection circuit, 208: image processing unit

Claims

1. An acquisition means for acquiring information on a gaze position of a user viewing a captured image; an estimation means for estimating a subject at which the user's gaze is directed based on the user's gaze position; a control means for controlling whether or not to display a first item indicating the subject estimated by the estimation means, based on at least one of a capture situation of the image and a situation of the subject included in the image; 1. An electronic device comprising:

2. The control means controls so as to display a second item indicating the user's gaze position when the user's gaze position is located away from a subject included in the image.

2. The electronic device according to claim 1 .

3. The control means When the user is paying attention to the subject estimated by the estimation means, When the imaging range of the image changes, or When the attitude of the imaging device that captures the image changes, Controlling to display the first item 2. The electronic device according to claim 1 .

4. The control means determines that the user is paying attention to the subject estimated by the estimation means when an amount of movement of the user's gaze position in a predetermined time is smaller than a first threshold value, or when a number of blinks in a predetermined time is smaller than a second threshold value.

4. The electronic device according to claim 3.

5. The control means determines that an imaging range of the image has changed when a zoom operation is performed, when an amount of change in focal length due to the zoom operation is greater than a third threshold value, or when an amount of change in zoom magnification due to the zoom operation is greater than a fourth threshold value.

4. The electronic device according to claim 3.

6. The control means determines that the imaging range of the image has changed when a panning operation or a tilting operation is performed, or when a movement amount of the imaging range due to the panning operation or the tilting operation is greater than a fifth threshold value.

4. The electronic device according to claim 3.

7. The control means determines whether or not the attitude of the imaging device has changed in a state in which information on the user's gaze position has been acquired.

4. The electronic device according to claim 3.

8. The control means When the number of objects included in the image changes, When the number of subjects included in the image is two or more, When the subject estimated by the estimation means is accelerating or decelerating, or When the subject estimated by the estimation means is not in focus, Controlling to display the first item 2. The electronic device according to claim 1 .

9. The number of subjects is the number of subjects of a predetermined type among the subjects included in the image.

9. The electronic device according to claim 8.

10. The number of subjects is the number of subjects detected within a predetermined range including the subject estimated by the estimation means.

9. The electronic device according to claim 8.

11. The predetermined range is determined based on at least one of the focal length, shooting distance, shooting mode information, and the size of the subject estimated by the estimation means when the image is captured.

11. The electronic device according to claim 10.

12. The control means performs control so as to display the first item when the subject estimated by the estimation means is in a focused state and a gaze position of the user is detected with respect to a subject different from the focused subject.

9. The electronic device according to claim 8.

13. When the control means fails to acquire information on the user's gaze position, If the first item has not been displayed in an image captured before the image, the first item is not displayed in the image; When the first item is displayed in an image captured before the image, the first item is displayed in the image at the same position as in the image captured before the image. Control so that 2. The electronic device according to claim 1 .

14. When the control means fails to acquire information on the user's gaze position and displays the first item on the image, the control means displays the first item in a manner different from a manner in which the control means successfully acquires information on the user's gaze position.

14. The electronic device according to claim 13.

15. An acquisition step of acquiring information on a gaze position of a user viewing a captured image; an estimation step of estimating a subject at which the user's gaze is directed based on the user's gaze position; a control step of controlling whether or not to display a first item indicating the subject estimated in the estimation step, based on at least one of a capture situation of the image and a situation of the subject included in the image; 13. A method for controlling an electronic device comprising:

16. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Equipment provided with line-of-sight detecting means

    JP1997015483A

  • Electronic apparatus

    JP2021131441A