Electronic apparatus and electronic apparatus control method

The electronic device enhances visibility by controlling the display of line-of-sight position and captured images based on gaze information, addressing visibility issues due to unsynchronized changes in display and subject situations.

JP2024170159APending Publication Date: 2024-12-06CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023087166
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The visibility of the user's line-of-sight position and captured image deteriorates when the display state and form of the line-of-sight position are changed without considering the imaging situation and subject situation.

Method used

An electronic device that acquires information on the user's gaze position and controls the display of items based on this position, transitioning between display and non-display states using various methods.

Benefits of technology

Improves the visibility of the line-of-sight position and captured image by appropriately transitioning between display states, ensuring clear visibility of the subject of interest.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024170159000001_ABST
    Figure 2024170159000001_ABST
Patent Text Reader

Abstract

To provide an electronic apparatus which properly transits a display state and a non-display state of a visual line position and thereby improves the visibility of the visual line position and a captured image.SOLUTION: The electronic apparatus comprises: acquisition means which acquires information on the visual line position of a user viewing a captured image; and control means which performs control to display an item at a position based on the visual line position of the user. The control means transits one of a display state and a non-display state of the visual line position to the other, by a transition system selected out of a plurality of transition systems.SELECTED DRAWING: Figure 8D
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an electronic device and a method for controlling an electronic device. [Background technology]

[0002] Conventionally, imaging devices equipped with a function for detecting the gaze position of a user looking into the viewfinder field of view have been provided. In recent years, imaging devices with electronic viewfinders capable of displaying images acquired by an imaging element that receives a light beam that has passed through a photographing optical system have been provided. Imaging devices with electronic viewfinders are capable of recognizing and detecting various types of subjects. Such imaging devices are capable of performing focus detection control at a subject position that reflects the user's intention, based on information on the detected gaze position and the recognition result of the subject in the acquired image.

[0003] Patent Document 1 discloses a technique for appropriately controlling the display state of the gaze position. Patent Document 2 discloses a technique for displaying the gaze position during focus detection and photometry processing and the gaze position immediately before shooting in different display forms. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-131441 A [Patent Document 2] Japanese Patent Application Publication No. 8-184746 Summary of the Invention [Problem to be solved by the invention]

[0005] However, if the display state and display form of the user's gaze position are changed without taking into consideration the imaging conditions and the conditions of the captured subject, the gaze position and the visibility of the captured image may be reduced.

[0006] Therefore, an object of the present invention is to provide an electronic device that improves the visibility of the gaze position and a captured image by appropriately transitioning between a display state and a non-display state of the gaze position. [Means for solving the problem]

[0007] A first aspect of the present invention is an electronic device having an acquisition means for acquiring information on the gaze position of a user viewing a captured image, and a control means for controlling the display of an item at a position based on the user's gaze position, wherein the control means transitions the item from one of a displayed state and a hidden state to the other in a transition method selected from a plurality of transition methods.

[0008] A second aspect of the present invention is a control method for an electronic device, comprising an acquisition step of acquiring information on the gaze position of a user viewing a captured image, and a control step of controlling to display an item at a position based on the user's gaze position, wherein the control step transitions the item from one of a displayed state and a hidden state to the other using a transition method selected from a plurality of transition methods.

[0009] A third aspect of the present invention is a program for causing a computer to function as each of the means of the electronic device described above. A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device described above. It is a medium. Effect of the Invention

[0010] According to the present invention, it is possible to provide an electronic device that improves the visibility of the gaze position and a captured image by appropriately transitioning between a display state and a non-display state of the gaze position. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the appearance of a camera. [Diagram 2]FIG. 2 is a diagram illustrating an example of the internal configuration of a camera. [Diagram 3] FIG. 2 is a block diagram showing an example of an electrical configuration within the camera. [Figure 4] FIG. 4 is a diagram showing an example of a display in a viewfinder field of view. [Diagram 5] 1 is a diagram for explaining the principle of a gaze detection method. [Figure 6] FIG. 2 shows an eye image. [Figure 7] 13 is a flowchart of a gaze detection process. [Figure 8A] 11 is a flowchart illustrating an icon display control process. [Figure 8B] 13 is a flowchart of an icon display determination process based on an imaging situation. [Figure 8C] 13 is a flowchart of an icon display determination process based on a subject situation. [Figure 8D] 4 is a flowchart of an icon display process according to the first embodiment. [Figure 8E] 11 is a flowchart of icon non-display processing according to the first embodiment. [Figure 9] FIG. 4 is a diagram illustrating subject movement information. [Figure 10] FIG. 13 is a diagram illustrating an icon indicating a gaze position. [Figure 11] FIG. 11 is a diagram illustrating icon display based on a zoom operation. [Figure 12] 11A and 11B are diagrams illustrating icon display based on changes in camera attitude. [Figure 13] 11A and 11B are diagrams illustrating icon display based on a change in the number of subjects. [Figure 14] 11A and 11B are diagrams illustrating icon display based on subject movement information. [Figure 15] 13A and 13B are diagrams illustrating an example in which an icon is displayed by changing the transparency. [Figure 16] FIG. 13 is a diagram showing an example of changing the size of an icon to a displayed state. [Figure 17A]10 is a flowchart of an icon display process according to the second embodiment. [Figure 17B] 13 is a flowchart of icon non-display processing according to the second embodiment. [Figure 18] FIG. 13 is a diagram illustrating icon display for multiple subjects that are close to each other. [Figure 19] FIG. 13 is a diagram illustrating icon display for multiple subjects that are not in close proximity to each other. [Figure 20A] 13 is a flowchart of an icon display process according to the third embodiment. [Figure 20B] 13 is a flowchart of icon non-display processing according to the third embodiment. [Figure 21] FIG. 13 is an external view of another electronic device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Embodiment 1] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0013] <Configuration Description> An example of the camera configuration will be described with reference to Figs. 1(a), 1(b) to 3. Figs. 1(a) and 1(b) show an example of the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to an embodiment. Fig. 1(a) is a front perspective view, and Fig. 1(b) is a rear perspective view. As shown in Fig. 1(a), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operating member capable of receiving a photographing operation from a user (photographer), is disposed on the camera housing 1B.

[0014] As shown in FIG. 1(b), an eyepiece lens 12 (eyepiece optical system) is disposed on the rear surface of the camera housing 1B, through which the user looks to view a display device 10 (display panel) contained within the camera housing 1B. The eyepiece optical system may include multiple lenses. Also arranged on the rear surface of camera housing 1B are operation members 41-43 capable of receiving various operations from the user. For example, operation member 41 is a touch panel capable of receiving touch operations, operation member 42 is an operation lever that can be pushed down in each direction, and operation member 43 is a four-way key that can be pushed in each of four directions. Operation member 41 (touch panel) is equipped with a display panel such as a liquid crystal panel, and has a function of displaying images on the display panel.

[0015] Fig. 2 is a cross-sectional view of camera 1 cut along the YZ plane defined by the Y axis and Z axis shown in Fig. 1(a), and shows an example of the internal configuration of camera 1. In Fig. 2, the same components as those in Figs. 1(a) and 1(b) are denoted by the same reference numerals.

[0016] Photographing lens unit 1A includes two lenses 101, 102, an aperture 111, an aperture drive section 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. Photographing lens unit 1A also includes an acceleration sensor 119, a zoom drive section 120, and a lens CPU 121. Note that for simplicity, two lenses 101, 102 are shown, but photographing lens unit 1A actually includes more than two lenses. Each component included in photographing lens unit 1A is controlled by lens CPU 121.

[0017] Aperture 111 provided in photographing lens unit 1A is driven by aperture drive section 112. Lens drive member 114 includes a drive gear and the like. Photocoupler 115 detects the rotation of pulse plate 116 linked to lens drive member 114 and transmits it to focus adjustment circuit 118. Focus adjustment circuit 118 drives lens drive motor 113 based on information from photocoupler 115 and information from camera housing 1B (information on lens drive amount), and moves lens 101 to change the focus position. Mount contact 117 is an interface between photographing lens unit 1A and camera housing 1B.

[0018] Acceleration sensor 119 is built into photographing lens unit 1A and detects panning of lenses 101, 102. Acceleration sensor 6 is built into camera housing 1B and detects panning of camera housing 1B. Panning of camera 1 is determined using at least one of acceleration sensor 119 built into photographing lens unit 1A and acceleration sensor 6 built into camera housing 1B.

[0019] The zoom driving unit 120 performs zoom driving based on a user's manual operation of an operating member provided on a lens barrel (not shown), or performs zoom driving based on an instruction from the camera housing 1B notified via the mount contact 117. The lens CPU 121 notifies the camera housing 1B of zoom driving information from the photographing lens unit 1A via the mount contact 117. The camera 1 can detect a change in the angle of view due to the zoom operation based on the zoom driving information.

[0020] The camera housing 1B includes an imaging element 2, a CPU 3, a memory section 4, a display device 10, a display device drive circuit 11, and the like. The imaging element 2 is disposed at a planned imaging surface of the photographing lens unit 1A. The CPU 3 is a central processing section of a microcomputer, and controls the entire camera 1. The memory section 4 stores images captured by the imaging element 2, and the like. The display device 10 is composed of a liquid crystal or the like, and displays the captured image (subject image), and the like, on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. A user can view an image (such as an image captured by the imaging element 2) displayed on the display surface of the display device 10 through an eyepiece 12.

[0021] The camera housing 1B includes an eyepiece lens 12, light sources 13a and 13b, a light splitter 15, and a light receiving lens. The light sources 13a and 13b include an optical system including an optical fiber 13B, an optical fiber 13C for detecting the direction of the line of sight (direction in which the user is looking) from the relationship between the image of the light reflected by the cornea (corneal reflection image; Purkinje image) and the pupil. Specifically, the light sources 13a and 13b are infrared light-emitting diodes or the like that emit infrared light that is insensitive to the user, and are arranged around the eyepiece 12.

[0022] The illuminated optical image of eyeball 14 (eye image; an image formed by light emitted from light sources 13a, 13b and reflected by eyeball 14) passes through eyepiece lens 12 and is reflected by light splitter 15. The eye image is then formed by light receiving lens 16 on ocular imaging element 17, which is a two-dimensional array of photoelectric elements such as CCD or CMOS. Light receiving lens 16 positions the pupil of eyeball 14 and ocular imaging element 17 in a conjugate imaging relationship. The line of sight of eyeball 14 is detected by a predetermined algorithm from the positional relationship between the pupil and the corneal reflection image in the eye image formed on ocular imaging element 17.

[0023] Fig. 3 is a block diagram showing an example of an electrical configuration in the camera 1. In Fig. 3, the same components as those in Figs. 1(a), 1(b) and 2 are denoted with the same reference numerals, and detailed description of these components will be omitted. The CPU 3 is connected to a line of sight detection circuit 201, a photometry circuit 202, an auto focus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, an acceleration sensor 6, a memory unit 4, an image processing unit 208 and an attitude detection unit 207.

[0024] Furthermore, CPU 3 transmits a signal to lens CPU 121 via mount contact 117. Lens CPU 121 controls focus adjustment circuit 118 arranged in photographing lens unit 1A, aperture control circuit 206 included in aperture drive section 112, acceleration sensor 119, and zoom drive section 120 based on the signal from CPU 3.

[0025] The memory unit 4 associated with the CPU 3 has a function of storing image signals from the image sensor 2 and the eye image sensor 17. The CPU 3 converts the image signal of the image sensor 2 stored in the memory unit 4 into a photographed image, and transfers the converted photographed image to a recording medium 7 connected to the memory unit 4.

[0026] The attitude detection unit 207 detects attitude information of the user holding the camera 1. The attitude detection unit 207 can detect the attitude information of the user using, for example, the acceleration sensor 6. The camera 1 can capture images in a normal position (horizontal position) in which the image sensor 2 is horizontal to the subject, and in a vertical position in which the image sensor 2 is vertical to the subject. The attitude detection unit 207 can detect a change in the attitude of the camera 1 even if the user changes the attitude of the camera 1 while looking through the viewfinder.

[0027] The gaze detection circuit 201 A / D converts the output of the eye imaging element 17 (an image of the eye) when an eyeball image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points used for gaze detection from the eye image according to a predetermined algorithm, and calculates the user's viewpoint (gaze position; position where the gaze is directed; position where the user is looking) on ​​the display surface of the display device 10 from the positions of the feature points.

[0028] The photometry circuit 202 amplifies, logarithmically compresses, and A / D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor; specifically, the luminance signal corresponding to the brightness of the field, and sends the resulting field luminance information to the CPU 3. The field luminance information is information that indicates the luminance of each pixel in the captured image. The photometry circuit 202 also detects a specific subject (specific object) from the image captured by the image sensor 2, and sends the resulting subject information and subject area information to the CPU 3. The subject information (object information) is information related to the detected subject, The subject area information (object area information) is information that indicates the area of ​​the detected subject. The subject information and the subject area information can also be obtained by analyzing the field luminance information.

[0029] The auto focus detection circuit 203 A / D converts signal voltages from a plurality of detection elements (a plurality of pixels) included in the image sensor 2 and used for phase difference detection, and sends the converted signal to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals from the plurality of detection elements. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are focus detection points at each of 180 locations on the image plane corresponding to the field of view in the viewfinder (the field of view when looking through the viewfinder) shown in FIG. 4(a), specifically, the 180 locations shown on the display surface of the display device 10.

[0030] Switches SW1 and SW2 are connected to the signal input circuit 204. Switch SW1 is turned on by the first stroke (half press) of the release button 5, and is a switch for starting the photometry and focus detection (distance measurement) of the camera 1. Switch SW2 is turned on by the second stroke (full press) of the release button 5, and is a switch for starting the photographing operation. ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3. Photometry and focus detection are performed within a set photometry area or focus detection area, or within the area of ​​a selected (determined) main subject. For example, in photometry, the photometry circuit 202 determines the luminance of the target area from field luminance information, and in focus detection, the auto focus detection circuit 203 determines the distance to a subject present in the target area.

[0031] An operation member 41 (touch panel), an operation member 42 (operation lever), and an operation member 43 (four-way key) send operation signals corresponding to operations received from a user to the CPU 3. A light source drive circuit 205 drives the light sources 13a and 13b.

[0032] The image processing unit 208 applies various types of image processing to the image captured by the imaging element 2. The image processing unit 208 also acquires and generates various types of information from the captured image. The image processing unit 208 may be realized by a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. The image processing unit 208 may also be realized by a processor such as a DSP (Digital Signal Processor) that realizes a specific function by executing software.

[0033] Image processing by the image processing unit 208 includes pre-processing, color interpolation, correction, detection, data processing, evaluation value calculation, etc. Pre-processing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation is a process of interpolating values ​​of color components not included in image data, and is also called demosaic processing. Correction processing includes a process of adjusting white balance, a process of correcting image luminance, a process of correcting optical aberration of the photographing lens unit 1A, a process of correcting color, etc.

[0034] The detection process includes detection of characteristic regions (e.g., regions of faces, human bodies, animals, automobiles, etc.), tracking of characteristic regions, person recognition, etc. The image processing unit 208 detects information on the user's gaze position on the display surface of the display device 10 that displays images, and detects the subject in the image, thereby making it possible to identify the subject that the user is paying attention to. In addition, the image processing unit 208 can obtain motion characteristic information of the subject, such as how the subject in the image is moving on the image plane, by analyzing the difference between images along a time series.

[0035] The data processing includes scaling, encoding and decoding, header information generation, etc. The evaluation value calculation process is a process for calculating a pair of signals for AF processing using a phase difference signal, a control signal, and a phase difference signal. This includes calculation of an evaluation value for last AF, an evaluation value used for automatic exposure control, etc. Note that the image processing performed by the image processing unit 208 is not limited to the above example.

[0036] 4(a) and 4(b) are diagrams showing examples of the display of the field of view in the finder, showing a state in which the display device 10 is operating (a state in which an image is displayed). As shown in FIG. 4(a), the field of view in the finder includes a field of view mask 300, a focus detection area 400, and 180 ranging point indices 401. Each of the 180 ranging point indices 401 is displayed superimposed on a through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. The image processing unit 208 can obtain information on the user's gaze position on the display surface of the display device 10 from an eye image obtained by the eye imaging element 17 when the user looks through the finder.

[0037] In FIG. 4(a), the gaze position 421 is the position (area including the gaze position) where the gaze of the user is detected. The subject 422 is a person detected by the image analysis by the image processing unit 208. In the following description, when the detection target is a person, the hatched head area is detected as the subject. Since a movement accompanied by a fluctuation called fixation eye movement occurs in the human eyeball, it is preferable that the image processing unit 208 estimates a subject existing near the gaze position as a subject to which the gaze of the user is directed. When there is a deviation between the gaze position and the position where the subject exists, the image processing unit 208 displays an icon 423 indicating the subject to which the gaze of the user is directed. By displaying the icon 423, the user can confirm the subject that is estimated to be looked at by the user. Furthermore, the image processing unit 208 can estimate the subject that the user is looking at by directing the gaze of the user to the vicinity of the desired subject.

[0038] The vicinity of the gaze position can be a predetermined range including the gaze position. The predetermined range is determined based on, for example, the focal length, shooting distance, shooting mode, and the size of the subject to which the gaze is directed (present at a position based on the user's gaze position) when capturing an image. The size of the subject is the size in the image, and can be expressed by, for example, the number of pixels occupied by the subject. The predetermined range may also be determined based on the type of subject, the moving speed, and the like. When multiple subjects are detected near the gaze position, the image processing unit 208 can estimate that the subject closest to the gaze position is the subject to which the user's gaze is directed.

[0039] Fig. 4(b) shows an example in which multiple subjects exist in an image. In the image shown in Fig. 4(b), subjects 431 and 432 are detected. Gaze position 433 is the position where the user's gaze is detected. Icon 434 (corresponding to a first item) shows the subject (hereinafter also referred to as an estimated subject) that is estimated by image processing unit 208 to be directed at by the user's gaze.

[0040] When the user moves the line of sight to the vicinity of the subject 432, the image processing unit 208 changes the estimated subject from the subject 431 to the subject 432. The image processing unit 208 displays an icon 434 indicating the estimated subject for the subject 432, which is the destination of the line of sight. Note that the subject detected in the image is not limited to a person, and may be an animal, a vehicle, or the like.

[0041] <Explanation of gaze detection process> The gaze detection process (gaze detection method) for acquiring the gaze position will be described with reference to Figures 5, 6(a), 6(b) and 7. The gaze of both the right eye and the left eye is detected by the following gaze detection method. Figure 5 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for detecting the gaze. As shown in Figure 5, light sources 13a and 13b are disposed approximately symmetrically with respect to the optical axis of light receiving lens 16, and illuminate the user's eyeball 14. A portion of the light emitted from light sources 13a and 13b and reflected by eyeball 14 is focused by light receiving lens 16 onto eye imaging element 17. Fig. 6(a) is a schematic diagram of an eye image captured by eye imaging element 17 (optical image of the eye projected onto eye imaging element 17), and Fig. 6(b) is a diagram showing the output intensity of eye imaging element 17. Fig. 7 is a flowchart of the gaze detection process.

[0042] 7 starts, in step S1, CPU 3 drives light sources 13a and 13b using light source drive circuit 205 so as to emit infrared light toward user's eyeball 14. An optical image of the user's eye illuminated by infrared light passes through light receiving lens 16 and is formed on eye imaging element 17, which then performs photoelectric conversion. This provides an electrical signal of the eye image that can be processed.

[0043] In step S2, the CPU 3 acquires an eye image (image data, image signal) from the eye imaging element 17 via the line of sight detection circuit 201.

[0044] In step S3, the CPU 3 acquires the corneal reflection images Pd, Pe of the light sources 13a, 13b and the coordinates of a point corresponding to the center c of the pupil 141 from the eye image obtained in S2. The infrared light emitted from the light sources 13a, 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflection images Pd, Pe formed by a part of the infrared light reflected on the surface of the cornea 142 are collected by the light receiving lens 16 and imaged on the eye imaging element 17 to become the corneal reflection images Pd', Pe' in the eye image. Similarly, the light beams from the ends a, b of the pupil 141 are also imaged on the eye imaging element 17 to become the pupil end images a', b' in the eye image.

[0045] Figure 6(b) shows luminance information (luminance distribution) of region α in the eye image of Figure 6(a). In Figure 6(b), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. The coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe, and the coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb.

[0046] As shown in FIG. 6(b), at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe', an extremely high level of luminance is obtained. In the region from the coordinates Xa to the coordinates Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the ocular imaging element 17), an extremely low level of luminance is obtained except for the coordinates Xd and Xe. And in the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 143), an intermediate luminance between the above two types of luminance is obtained. For example, an intermediate luminance between the above two types of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is greater than the coordinate Xa and the region where the X coordinate is less than the coordinate Xb.

[0047] From the luminance distribution as shown in FIG. 6(b), the X coordinates Xd, Xe of the corneal reflection images Pd', Pe' and the X coordinates Xa, Xb of the pupil edge images a', b' can be obtained. For example, the coordinates of the corneal reflection images Pd', Pe' can be obtained as the coordinates of the corneal reflection images, and the coordinates of the pupil edge images a', b' can be obtained as the coordinates of the pupil edge images a', b'. In addition, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by forming an image of the light beam from the pupil center c on the ocular imaging element 17 can be expressed as Xc≒(Xa+Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this manner, the coordinates of the corneal reflection images Pd', Pe' and the coordinates of the pupil center image c' can be estimated.

[0048] In step S4, CPU 3 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of eyeball 14 relative to light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between corneal reflection images Pd', Pe'.

[0049] In step S5, the CPU 3 determines the rotation of the optical axis of the eyeball 140 relative to the optical axis of the light receiving lens 16. The rotation angle is calculated. The X coordinate of the midpoint between the corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of the cornea 142 are almost the same. Therefore, if the standard distance from the center of curvature O of the cornea 142 to the center of the pupil c is Oc, the rotation angle θ of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) is X can be calculated by the following Equation 1. The rotation angle θy of the eyeball 140 in the ZY plane (plane perpendicular to the X-axis) can also be calculated in a similar manner to the calculation of the rotation angle θx. β*Oc*SINθ X ≒{(Xd + Xe) / 2}-Xc (Eq. 1)

[0050] In step S6, the CPU 3 uses the rotation angles θx, θy calculated in step S5 to estimate the user's line of sight (hereinafter also referred to as the viewpoint) on the display device 10. If the coordinates (Hx, Hy) of the viewpoint correspond to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0051] Parameter m in Equations 2 and 3 is a constant determined by the configuration of the optical system for performing gaze detection processing, and is a conversion coefficient for converting rotation angles θx, θy into coordinates corresponding to the pupil center c on the display device 10. Parameter m is determined in advance and stored in the memory unit 4. Gaze correction parameters Ax, Bx, Ay, By are gaze correction parameters that correct individual differences in gaze. The gaze correction parameters Ax, Bx, Ay, By are acquired by performing calibration of gaze detection, and are stored in advance in the memory unit 4.

[0052] In step S7, CPU 3 stores the coordinates of the viewpoint and the time when the eye image was acquired (hereinafter, gaze detection time) in memory unit 4, and ends the gaze detection process.

[0053] The gaze detection method is not limited to the above method, and may be any method that detects the gaze based on an eye image. In addition, although an example in which the coordinates (Hx, Hy) of the viewpoint are obtained as the result (final result) of gaze detection has been described, any information related to the gaze (gaze information), such as rotation angles θx, θy, may be obtained as the result of gaze detection.

[0054] <Description of icon display control process> 8A to 8C, the display control of an icon indicating a subject at which a user's gaze is directed will be described. Fig. 8A is a flowchart illustrating an icon display control process in the first embodiment. The process shown in Fig. 8A includes image acquisition, subject detection, and icon display control processes. CPU 3 acquires images in chronological order, and repeatedly executes the process shown in Fig. 8A.

[0055] In step S8001, CPU 3 acquires an image captured by imaging element 2 and generated by image processor 208 through various image processing. In step S8002, CPU 3 detects a subject included in the image acquired in step S8001. Specifically, CPU 3 acquires information on the position of the head of a person detected from the image as subject position information. For example, in the image of FIG. 9(a), CPU 3 detects subject 901 and acquires information on the position of the head of subject 901.

[0056] In step S8003, the CPU 3 acquires information on the movement of the subject included in the image (information on the movement characteristics, information on the action). The CPU 3 can acquire information on the movement of the subject included in the image by comparing the image acquired in step S8001 with an image previously generated by the image processing unit 208. For example, in FIGS. 9(a) to 9(c), at time t1, These are images generated at time t2, which is later than time t1, and at time t3, which is later than time t2. In this case, CPU 3 can obtain motion information indicating that subject 901 is moving at a substantially constant speed. Also, as shown in FIG. 9(d), when multiple subjects including subject 901 are detected in the image and all show similar motion characteristics, these multiple subjects are treated as subject group 902.

[0057] In step S8004, CPU 3 performs the gaze detection process described in Figs. 5 to 7 using eye imaging element 17 and gaze detection circuit 201 to obtain gaze position information. In step S8005, CPU 3 determines whether gaze detection (obtaining gaze position information) in step S8004 was successful. If gaze detection was successful, the process proceeds to step S8007. If gaze detection was unsuccessful, the process proceeds to step S8006.

[0058] In step S8006, CPU 3 maintains the display position of the icon (first item) in the previous process (processing for the image captured before the current image) and the display state of whether to display it, and returns to step S8001. Specifically, if the icon was not displayed in the previous process, CPU 3 controls not to display the icon for the current image either, and if the icon was displayed in the previous process, CPU 3 controls to display the icon for the current image in the same position as the previous image.

[0059] If the gaze detection fails, the CPU 3 may display the icon in a different manner from when the gaze detection succeeds in order to inform the user that the gaze detection has failed. For example, if the gaze detection fails, the CPU 3 may display the icon in a different color from when the gaze detection succeeds, or may display the icon in a blinking manner. Note that, in order to prevent the visibility from decreasing due to frequent changes in the color of the icon, the CPU 3 may change the display manner of the icon after a predetermined time has elapsed since it was determined that the gaze detection has failed.

[0060] In step S8007, CPU 3 determines whether or not a subject is detected at the gaze position acquired in step S8004. When a subject detected in step S8002 exists near the user's gaze position, CPU 3 can determine that a subject is detected at the gaze position. If a subject is detected at the gaze position, processing proceeds to step S8009. If a subject is not detected at the gaze position, processing proceeds to step S8008.

[0061] In step S8008, CPU 3 displays an icon at the gaze position acquired in step S8004. For example, in Fig. 10, the user's gaze position is located away from subject 1001 included in the image, so CPU 3 displays icon 1002 (second item) indicating the detected user's gaze position. For example, when subject 1001 included in the image is away from the user's gaze position by a predetermined distance or more, CPU 3 displays an icon indicating the user's gaze position.

[0062] When displaying icon 1002 indicating the gaze position, since there is no subject at the user's gaze position, CPU 3 does not display an icon (first item) indicating the subject at which the user's gaze is directed. This allows CPU 3 to inform the user that subject 1001 is not captured as the subject at which the user's gaze is directed.

[0063] Area 1003 is an area indicating the vicinity of the user's gaze position. CPU 3 sets area 1003 as an area for determining whether or not a subject is present in the vicinity of the gaze position. Area 1003 may be set by the user using operation members 41 to 43. CPU 3 may also set area 1003 automatically in accordance with the optical state of the lens, such as the focal length of photographing lens unit 1A or the photographing object distance at the in-focus position, or the area set by camera 1. Alternatively, the area 1003 may be set automatically according to the mode setting of the AF function, etc. In other words, the vicinity of the gaze position is an area that includes the gaze position of the user and can be set by the user or the CPU 3.

[0064] In step S8009, CPU 3 determines whether to display an icon indicating a subject at which the user's line of sight is directed, based on the status of the user's image capture with camera 1. The status of the image capture with camera 1 includes, for example, statuses such as whether the user is paying attention to the subject estimated by image processing unit 208, whether the imaging range of the image has changed, and whether the attitude of the imaging device capturing the image has changed.

[0065] Details of the process of step S8009 will be described with reference to Fig. 8B. In step S8201, CPU 3 determines whether or not the user, who is the photographer, is paying attention to a subject that he or she wishes to photograph (a subject that is estimated to be directed at the user's gaze, an estimated subject).

[0066] Whether the user is paying attention to the estimated subject is determined from the eye image captured by the eye image sensor 17 based on the degree of stability of the user's gaze position, the presence or absence or frequency of the user's blinking, etc. For example, the CPU 3 can determine that the user is paying attention to the estimated subject when the amount of movement of the user's gaze position in a predetermined time is smaller than a first threshold value, or when the number of blinks in a predetermined time is smaller than a second threshold value. Since the degree of stability of the gaze position and the frequency of blinking vary from person to person, the first threshold value and the second threshold value may be set in advance for each user. If the user is paying attention to the estimated subject, the process proceeds to step S8202. If the user is not paying attention to the estimated subject, the process proceeds to step S8203.

[0067] In step S8202, CPU3 determines to display an icon indicating the estimated subject. By displaying the icon indicating the estimated subject, CPU3 can inform the user that the subject the user is paying attention to is captured as a subject to be captured. For example, when the user is waiting for the subject he / she wants to photograph at a dolphin show to move to a desired position, or when the user is waiting for a photo opportunity in front of a soccer goal, the icon indicating the subject the user is paying attention to is displayed, so that the user does not miss the photo opportunity.

[0068] In step S8203, CPU 3 determines whether or not a zoom operation, pan operation, or tilt operation has been performed. Zoom operation includes a user's zoom operation on photographing lens unit 1A and a digital zoom operation on camera 1. CPU 3 determines whether or not a zoom operation, pan operation, or tilt operation has been performed, thereby determining whether or not the imaging range of the image has changed.

[0069] For example, the CPU 3 can determine that the imaging range of the image has changed when a zoom operation is performed, when the amount of change in focal length due to the zoom operation is greater than a third threshold, or when the amount of change in zoom magnification due to the zoom operation is greater than a fourth threshold. The CPU 3 can also determine that the imaging range of the image has changed when a panning operation or a tilting operation is performed, or when the amount of movement of the imaging range due to the panning operation or the tilting operation is greater than a fifth threshold. The third to fifth thresholds can be determined based on, for example, whether or not there is a possibility that the subject on which the user is directing his or her gaze will be lost due to the change in the imaging range. If a zoom operation, a panning operation, or a tilting operation is performed, the process proceeds to step S8202. If a zoom operation, a panning operation, or a tilting operation is not performed, the process proceeds to step S8204.

[0070] In step S8202, the CPU 3 determines to "display an icon" indicating the estimated subject. When the composition is changed by a zoom operation or the like, the position of the subject on which the user was directing his or her gaze may change. If a different subject appears in the position of the subject on which the user was originally directing his or her gaze, If the subject becomes present, there is a possibility that the user will lose sight of the subject that he or she was trying to photograph. Therefore, when the imaging range of the image is changed by a zoom operation or the like, the CPU 3 displays an icon indicating the presumed subject, thereby preventing the user from losing sight of the subject that he or she was trying to photograph.

[0071] 11(a) and 11(b) are diagrams for explaining icon display based on a zoom operation. Fig. 11(a) shows a state in which a subject 1101 is estimated as a subject at which the user's gaze is directed when zoomed in to the TELE side. Since there is only one subject near gaze position 1102 (the icon indicating gaze position 1102 is not displayed on the display device 10), the icon indicating the subject at which the user's gaze is directed is not displayed.

[0072] Fig. 11(b) shows the situation after zoom operation in the WIDE direction from the state of Fig. 11(a). In Fig. 11(b), a subject 1103 different from subject 1101 is detected in the image. If the gaze position 1102 is the same as before the zoom operation, the user may be confused as to whether subject 1103 has been captured as an estimated subject. Therefore, when the user is directing his / her gaze at subject 1101, CPU 3 determines to display an icon for subject 1101 in order to inform the user that subject 1101 has been captured as an estimated subject.

[0073] In step S8204, CPU 3 determines whether the attitude of camera 1 has changed. When shooting with camera 1, the user alternates between a horizontal position where the image is shot so that the long side of the image is horizontal, and a vertical position where the image is shot so that the long side of the image is vertical. CPU 3 determines whether the attitude of camera 1 has changed, for example, by changing the camera from a horizontal position to a vertical position. CPU 3 can obtain information on the attitude of camera 1 using attitude detection unit 207.

[0074] CPU 3 may determine whether the attitude of camera 1 has changed after acquiring information on the user's gaze position. If the attitude of camera 1 is changed while looking through the viewfinder, an icon indicating the estimated subject is displayed, making it easier for the user to confirm the desired subject. If the attitude of camera 1 has changed, processing proceeds to step S8202. If the attitude of camera 1 has not changed, processing proceeds to step S8205.

[0075] In step S8202, CPU 3 determines to display an icon indicating the estimated subject. This is because when the attitude of camera 1 changes, the user's line of sight is likely to become unstable, and this is to prevent the user from losing sight of the subject that was being viewed before the attitude of camera 1 changed.

[0076] 12(a) and 12(b) are diagrams for explaining icon display based on a change in the posture of the camera 1. FIG. 12(a) shows an example of an image captured in a horizontal position. In FIG. 12(a), subjects 1201 and 1202 are detected. The user's gaze is directed toward the subject 1201. FIG. 12(b) shows an example of an image after the user changes the position of the camera 1 from the state shown in FIG. 12(a) to a vertical position. While the camera 1 is changed from the horizontal position to the vertical position, the position of the user's gaze becomes unstable due to the change in composition. To prevent the user from mistakingly recognizing the subject 1202 as the subject 1201 that he had been looking at, the CPU 3 determines to display an icon 1203 indicating the estimated subject.

[0077] In step S8205, the CPU 3 determines that it is undecided whether or not to display an icon indicating the estimated subject ("icon display undecided"). In other words, if none of the conditions in steps S8201, S8203, and S8204 is satisfied, the CPU 3 determines in step S8020 of FIG. 8A described later that the icon indicating the estimated subject is displayed based on the state of the subject. Determine whether to display the

[0078] In step S8206, CPU 3 determines whether the icon indicating the estimated subject is currently in a display state or a non-display state. If the icon indicating the estimated subject is in a non-display state, the process proceeds to step S8207. If the icon indicating the estimated subject is in a display state, the process shown in FIG. 8B ends.

[0079] In step S8207, CPU 3 records the current time as a reference time in memory unit 4. Step S8207 is a process performed in the case where the icon indicating the estimated subject is currently in a non-display state and it is determined in step S8202 that the icon is to be displayed. By recording the time (reference time) at which it is determined that the icon will transition from a non-display state to a display state, CPU 3 can measure the time since the icon transitioned to a display state.

[0080] As described above, by the process shown in Fig. 8B, the CPU 3 determines whether or not to display an icon of a subject at which the user's gaze is directed, based on the imaging situation (the situation of the photographer). The CPU 3 ends the process of Fig. 8B, and proceeds to step S8010 of Fig. 8A.

[0081] In step S8010 of Fig. 8A, CPU 3 determines whether or not the result of the display determination of the icon indicating the estimated subject in step S8009 was "display icon". If the determination result is "display icon", the process proceeds to step S8011. If the determination result is "icon display is undecided", the process proceeds to step S8020.

[0082] In step S8011, CPU 3 determines whether or not a predetermined time has elapsed since the reference time recorded in step S8207 in Fig. 8B. That is, CPU 3 determines whether or not a predetermined time has elapsed since the time (reference time) when the icon representing the estimated subject transitioned from a non-display state to a display state. If the predetermined time has elapsed since the reference time, the process proceeds to step S8013. If the predetermined time has not elapsed since the reference time, the process proceeds to step S8012.

[0083] In step S8012, CPU 3 displays an icon indicating the estimated subject (for example, icon 423 in FIG. 4(a)). In step S8013, CPU 3 hides the icon indicating the estimated subject. After a predetermined time has elapsed since the imaging situation changed and an icon indicating the estimated subject was displayed, it is expected that the user will be able to stably direct their gaze toward a desired subject, so CPU 3 hides the icon indicating the estimated subject.

[0084] Although an example has been shown in which the icon indicating the estimated subject is automatically hidden when a predetermined time has elapsed since the icon indicating the estimated subject was switched from a non-display state to a display state, the CPU 3 may maintain the display state of the icon. Also, the predetermined time from the reference time until the icon indicating the estimated subject is hidden may be set by the user. In steps S8012 and S8013, when the icon indicating the estimated subject is respectively switched to a display state and a non-display state, the process returns to step S8001.

[0085] In step 8020, CPU 3 determines whether to display an icon indicating the subject at which the user's gaze is directed, based on the subject's status. The subject's status includes, for example, whether the number of subjects included in the image has changed, whether the number of subjects included in the image is two or more, whether the estimated subject is accelerating or decelerating, and whether the estimated subject is in focus.

[0086] Details of the process of step S8020 will be described with reference to FIG. 8C. In step S8301, CPU 3 checks whether the number of subjects in an image has increased or decreased since the previous image (frame). It is determined whether the subject is a person, an animal, a car, or the like. The number of subjects is not limited to the number of subjects in the image, and may be the number of subjects detected in the vicinity of the gaze position. Specifically, the number of subjects may be the number of subjects detected in a predetermined range including the estimated subject to which the gaze is directed. The predetermined range may be determined based on at least one of the focal length, shooting distance, shooting mode, and size of the estimated subject when capturing the image, for example. The subject to be detected is a person, an animal, a car, or the like that can be detected by analyzing the image.

[0087] The CPU 3 may detect a predetermined type of subject from among the subjects included in the image. For example, if the user sets animals as the predetermined type, the CPU 3 detects animals as the subject. The user may set multiple types of subjects as the predetermined type. If there is an increase or decrease in the number of detected subjects, the process proceeds to S8302. If there is no increase or decrease in the number of detected subjects, the process proceeds to S8303.

[0088] In step S8302, CPU 3 determines to display an icon indicating the estimated subject. By displaying an icon indicating the estimated subject, even if the number of subjects increases or decreases, the user can easily confirm the subject that the user is looking at.

[0089] In step S8303, CPU 3 determines whether or not the number of subjects detected near the gaze position is 1. If the number of subjects is 1, the process proceeds to step S8304. If the number of subjects is more than one, the process proceeds to step S8307.

[0090] In step S8304, CPU 3 determines whether or not one object existing in the vicinity of the gaze position in the screen is in focus. If the object is in focus, the process proceeds to step S8305. If the object is not in focus, the process proceeds to step S8306.

[0091] In step S8305, CPU 3 determines to "hide the icon." In step S8306, CPU 3 determines to "display the icon." CPU 3 can notify the user that the subject at which the user is looking is not in focus by displaying the icon when the subject is out of focus.

[0092] 13(a) and 13(b) are diagrams illustrating icon display based on the change in the number of subjects described in the processing of steps S8301 to S8306. Fig. 13(a) shows a situation where, at time t11, subject 1301 is present near gaze position 1302. Since it is clear that the subject 1301 is the object toward which the user's gaze is directed, the icon indicating the estimated subject and the icon indicating gaze position 1302 are not displayed.

[0093] Fig. 13(b) shows a state in which a different subject 1303 has entered the composition after the situation in Fig. 13(a). Because subject 1303 has appeared near gaze position 1304, it is preferable for CPU 3 to notify the user of whether the gaze was directed toward subject 1301 or subject 1303. Therefore, CPU 3 displays icon 1305 indicating the subject toward which the user's gaze was directed. Note that the icon indicating gaze position 1304 is not displayed.

[0094] In step S8307, CPU 3 determines whether or not a subject present near the user's gaze position is stationary or moving at a constant speed. Step S8307 is a process executed when multiple subjects are present near the gaze position. CPU 3 acquires information on the movement characteristics of the subjects near the gaze position (movement information). The movement information can be acquired in the same manner as in step S8003 of FIG. 8A described using FIGS. 9(a) to 9(c). Depending on the movement characteristics (movement) of the subjects, it is preferable to display an icon indicating the subject at which the user's gaze is directed so that the user does not lose sight of the desired subject.

[0095] If CPU 3 determines based on the motion information that the subject present near the gaze position is stationary or moving at a constant speed, CPU 3 proceeds to step S8308. On the other hand, if CPU 3 determines that any of the subjects present near the gaze position is moving while accelerating or decelerating, CPU 3 proceeds to step S8310.

[0096] In step S8308, CPU 3 determines whether the subject the user is looking at is in focus. If the subject the user is looking at is in focus, the process proceeds to step S8309. If the subject the user is looking at is out of focus, the process proceeds to step S8310.

[0097] In step S8309, CPU 3 determines to “hide the icon.” When the subject is in focus, it is assumed that the user's line of sight is stably directed toward the desired subject, and therefore it is determined to hide the icon indicating the estimated subject.

[0098] In step S8310, CPU 3 determines to "display an icon." If the subject is out of focus, it is assumed that the user is not stable, and so it is determined to display an icon indicating the presumed subject. Also, if a subject present in the vicinity of the gaze position is moving while accelerating or decelerating (for example, when performing vigorous movements such as dancing), CPU 3 determines to display an icon indicating the subject at which the user is directing his or her gaze. By displaying an icon indicating the subject at which the user is directing his or her gaze, the user is less likely to lose sight of the desired subject.

[0099] 14(a) to 14(d) are diagrams for explaining icon display based on subject movement information described in the processing of steps S8307 to S8310. Fig. 14(a) is an example of so-called portrait photography, in which multiple subjects are stationary. Subject 1401 detected from the image is assumed to be out of focus. CPU 3 estimates that the user's gaze is directed at subject 1401 present in the vicinity of gaze position 1402, and displays icon 1403 indicating that subject 1401 is an estimated subject.

[0100] 14(b) shows a situation in which subject 1401 has entered an in-focus state. CPU 3 displays icon 1404 indicating that subject 1401 is in-focus, and does not display icon 1403 indicating that subject 1401 is an estimated subject. By preferentially displaying icon 1404 indicating the in-focus state, CPU 3 can reduce the amount of information to be displayed and ensure visibility. CPU 3 can also notify the user in a timely manner that subject 1401 has entered an in-focus state.

[0101] Fig. 14(c) shows a situation where a subject group 1414 including a subject 1411 near the gaze position and a plurality of surrounding subjects have similar movement characteristics. Fig. 14(c) shows the situation at time t21. CPU 3 displays icon 1413 indicating that subject 1411 near gaze position 1412 is an estimated subject.

[0102] Fig. 14(d) shows a situation after the situation in Fig. 14(c), where a group of subjects 1414 including subject 1411 and multiple surrounding subjects are moving in the same direction at a substantially constant speed. Fig. 14(d) shows the situation at time t22. At this time, a subject 1415 different from subject 1411 is present near gaze position 1412. Therefore, the subject at which the user's gaze is directed is estimated to be subject 1415.

[0103] In FIG. 14(d), it is assumed that the subject 1415 is in focus. At time t22, the subject 1411 and the subject 1415 included in the subject group 1414 are in focus due to similar movement characteristics. The subject 1415 moves in a focused state. The CPU 3 displays an icon 1416 indicating that the subject 1415 is in focus, but does not display an icon indicating that the subject is an estimated subject. The CPU 3 does not display an icon indicating that the subject 1415 is an estimated subject, and instead displays the icon 1416 preferentially, thereby suppressing the amount of information displayed and ensuring visibility. The CPU 3 can also notify the user in a timely manner that the subject 1415 has become in focus.

[0104] In step S8312, CPU 3 determines whether it has been determined in the processing of steps S8301 to S8310 that "icons are to be displayed" or that "icons are to be hidden." If it has been determined that "icons are to be displayed," the processing proceeds to step S8313. If it has been determined that "icons are to be hidden," the processing shown in FIG. 8C ends.

[0105] In step S8313, CPU 3 determines whether the icon indicating the estimated subject is currently in a display state or a non-display state. If the icon indicating the estimated subject is in a non-display state, the process proceeds to step S8314. If the icon indicating the estimated subject is in a display state, the process shown in FIG. 8C ends.

[0106] In step S8314, CPU 3 records the current time as a reference time in memory unit 4. Step S8314 is a process performed when the icon indicating the estimated subject is currently in a non-display state and it is determined that the icon is to be "displayed." By recording the time (reference time) at which it is determined that the icon will be switched from a non-display state to a display state, CPU 3 can measure the time since the icon transitioned to a display state.

[0107] As described above, by the process shown in Fig. 8C, CPU 3 determines whether or not to display an icon of a subject at which the user's gaze is directed, based on the state of the subject. CPU 3 ends the process in Fig. 8C, and proceeds to step S8021 in Fig. 8A.

[0108] In step S8021 of Fig. 8A, CPU 3 determines whether or not the result of the display determination of the icon indicating the estimated subject in step S8020 was "display the icon". If the determination result is "display the icon", the process proceeds to step S8022. If the determination result is "hide the icon", the process proceeds to step S8025.

[0109] In step S8022, CPU 3 determines whether or not a predetermined time has elapsed since the reference time recorded in step S8314 of Fig. 8C. That is, CPU 3 determines whether or not a predetermined time has elapsed since the time (reference time) when the icon representing the estimated subject transitioned from a non-display state to a display state. If the predetermined time has elapsed since the reference time, the process proceeds to step S8024. If the predetermined time has not elapsed since the reference time, the process proceeds to step S8023.

[0110] In step S8023, CPU 3 displays an icon indicating the subject (estimated subject) at which the user's gaze is directed. In step S8024, CPU 3 hides the icon indicating the estimated subject. When the icon indicating the estimated subject becomes a displayed state and a hidden state in steps S8023 and S8024, respectively, the process returns to step S8001.

[0111] In step S8025, CPU 3 hides the icon indicating the subject (estimated subject) at which the user's gaze is directed, in response to the determination result in step S8020. In step S8026, CPU 3 initializes the reference time recorded in step S8207 or step S8314, and returns to step S8001.

[0112] 8D and 8E, the display control of the icon (item) indicating the estimated subject is performed. In the process of Fig. 8D, the CPU 3 displays an icon indicating the estimated subject at a position based on the user's gaze position. The position based on the user's gaze position includes the position where the user's gaze is detected and the position of the subject detected in the vicinity of the user's gaze position.

[0113] The CPU 3 also transitions the icon representing the estimated subject from one of the display state and the non-display state to the other in a transition manner selected from a plurality of transition manners, including, for example, a method of gradually changing the size of the icon, a method of gradually changing the transparency of the icon, and a method of immediately transitioning the icon from one of the display state and the non-display state to the other.

[0114] Fig. 8D is a flowchart of an icon display process according to embodiment 1. The process in Fig. 8D is detailed processing of steps S8012 and S8023 in Fig. 8A. In the processing shown in Fig. 8D, the CPU 3 transitions an icon indicating a subject (estimated subject) at which the user's gaze is directed to a display state.

[0115] In step S8401, CPU 3 determines whether the icon indicating the estimated subject is currently in a hidden state. If the icon indicating the estimated subject is in a hidden state, the process proceeds to step S8402. If the icon indicating the estimated subject is in a displayed state, the process shown in FIG. 8D ends.

[0116] In step S8402, CPU 3 determines whether or not it was determined in step S8201 of Fig. 8B that "the subject is being watched." If it is determined that the subject is being watched, the process proceeds to S8403. If it is not determined that "the subject is being watched," the process proceeds to S8404.

[0117] In step S8403, CPU 3 transitions the icon to a display state by gradually changing the transparency or size of the icon indicating the estimated subject. In a transition method in which the transparency of the icon is gradually decreased, the icon is displayed to become gradually darker. In a transition method in which the size of the icon is gradually increased, the icon is displayed to become gradually larger. By gradually changing the transparency or size of the icon, it becomes easier for the user to recognize the position of the icon indicating the estimated subject.

[0118] Figures 15(a) to 15(d) are diagrams showing an example of changing the transparency of an icon to a display state, and Figures 16(a) to 16(d) are diagrams showing an example of changing the size of an icon to a display state.

[0119] In Fig. 15(a), a gaze position 1502 is detected with respect to a subject 1501. In Fig. 15(a), an icon indicating gaze position 1502 is hidden. In the situation of Fig. 15(a), when it is determined that the user is paying attention to the subject, CPU3 starts displaying icon 1503, and gradually decreases the transparency of icon 1503 indicating the estimated subject to make it darker, as shown in Figs. 15(b), 15(c), and 15(d). As the user continues to pay attention to the subject, CPU3 gradually increases the density of icon 1503 (decreases the transparency) in the order of Figs. 15(b), 15(c), and 15(d).

[0120] In Fig. 16(a), a gaze position 1602 is detected with respect to a subject 1601. In Fig. 16(a), an icon indicating gaze position 1602 is hidden. When it is determined that the user is paying attention to the subject in the situation of Fig. 16(a), CPU 3 starts displaying icon 1603, and gradually increases the size of icon 1603 indicating the estimated subject, as shown in Fig. 16(b), Fig. 16(c), and Fig. 16(d). As the user continues to pay attention to the subject, CPU 3 gradually increases the size of icon 1603 in the order of Fig. 16(b), Fig. 16(c), and Fig. 16(d).

[0121] Gradually changing the transparency or size of the icon (icon 1503, icon 1603) indicating the estimated subject to inform the user of its presence allows the user to concentrate on shooting without being carelessly distracted. Gradually changing the transparency or size of the icon also makes it easier for the user to recognize the position of the icon indicating the estimated subject.

[0122] If it is not determined in step S8402 that the subject is being watched, in step S8404, CPU 3 determines whether the subject was determined to be in a "focused state" in step S8304 or step S8308 in Fig. 8C. If it is determined to be in a "focused state", the process proceeds to S8406. If it is not determined to be in a "focused state", that is, if it is determined to be in an "unfocused state", the process proceeds to S8405.

[0123] In step S8405, CPU 3 gradually changes the transparency or size of the icon indicating the estimated subject, as in step S8403, to transition the icon to a display state. Note that CPU 3 may transition the icon indicating the estimated subject from a non-display state to a display state by different transition methods in steps S8403 and S8405. For example, if CPU 3 transitions the icon to a display state by changing the transparency in step S8403, CPU 3 transitions the icon to a display state by changing the size in step S8405. Conversely, if CPU 3 transitions the icon to a display state by changing the size in step S8403, CPU 3 may transition the icon to a display state by changing the transparency in step S8405.

[0124] In step S8406, CPU 3 places importance on responsiveness and immediately displays an icon indicating the estimated subject.

[0125] Fig. 8E is a flowchart of icon non-display processing according to embodiment 1. The processing in Fig. 8E is details of the processing in steps S8013, S8024, and S8025 in Fig. 8A. In the processing shown in Fig. 8E, the CPU 3 transitions the icon indicating the subject (estimated subject) at which the user's gaze is directed to a non-display state.

[0126] In step S8501, CPU 3 determines whether or not the icon representing the estimated subject is currently in a display state. If the icon representing the estimated subject is in a display state, the process proceeds to step S8502. If the icon representing the estimated subject is in a non-display state, the process shown in FIG. 8E ends.

[0127] In step S8502, CPU 3 determines whether or not a blink of the user was detected in step S8201 of FIG. 8B. That is, CPU 3 determines whether or not the icon is switched to hidden due to the user blinking. If a blink is detected, the process proceeds to step S8503. If a blink is not detected, the process proceeds to step S8505.

[0128] In step S8503, CPU 3 acquires blinking characteristic information of the user, specifically, information on the time period during which the user's eyes are closed while blinking. CPU 3 can acquire the information on the time period during which the user's eyes are closed while blinking, for example, by analyzing the difference between eye images along a time series using image processor 208.

[0129] In step S8504, the CPU 3 determines a predetermined transition time until the icon representing the estimated subject is hidden based on the blink characteristic information acquired in step S8503. The predetermined transition time is a time longer than the time during which the user closes his / her eyes while blinking. The CPU 3 gradually changes the transparency or size of the icon indicating the estimated subject over a predetermined transition time, thereby transitioning the icon to a non-display state.

[0130] It is assumed that the time for which the eyes are closed during blinking may differ depending on the degree of eye fatigue or may differ for each user. By transitioning the icon representing the estimated subject to a hidden state over a period of time (a predetermined transition time) longer than the time for which the user's eyes are closed during blinking, the user can recognize that the icon is gradually transitioning from a visible state to a hidden state.

[0131] In step S8505, CPU 3 determines whether or not the composition has been changed due to a zoom operation, a pan operation, or a tilt operation being performed in step S8203 of FIG. 8B, or due to a change in the attitude of camera 1 in step S8204. That is, CPU 3 determines whether or not the icon indicating the estimated subject has been displayed due to a change in the composition. If it is determined that the composition has been changed, the process proceeds to step S8506. If it is determined that the composition has not been changed, the process proceeds to step S8507.

[0132] In step S8506, CPU 3 gradually changes the transparency or size of the icon indicating the estimated subject, thereby transitioning the icon to a non-display state. Immediately after changing the composition, the user's gaze position may become unstable. In addition, the user may change the composition again after changing it. By gradually transitioning the icon from a display state to a non-display state, the user can check the gaze position until it is switched to a non-display state. Therefore, the operability for the user to determine the composition is improved.

[0133] In step S8507, CPU 3 determines whether or not there are multiple subjects detected near the gaze position, similarly to step S8303 in Fig. 8C. If there are multiple subjects detected near the gaze position, the process proceeds to step S8508. If there is only one subject detected near the gaze position, the process proceeds to step S8511.

[0134] In step S8508, CPU 3 determines whether the subject at the gaze position in the current image (the subject at which the user's gaze is directed, the estimated subject) is the same as the estimated subject in the previous image. That is, CPU 3 determines whether the subject the user is currently looking at is the same as the subject the user looked at in the past. For example, CPU 3 may determine whether the subject present at a position based on the user's current gaze position is the same as the subject that was present at a position based on the user's gaze position in an image (previous frame) captured before the current image (current frame). If the estimated subject is the same as the previous one, processing proceeds to step S8509. If the estimated subject is not the same as the previous one, processing proceeds to step S8510.

[0135] In step S8509, CPU 3 gradually changes the transparency of the icon indicating the estimated subject, thereby transitioning the icon to a non-display state. In step S8510, CPU 3 gradually reduces the size of the icon indicating the estimated subject, thereby transitioning the icon to a non-display state.

[0136] When transitioning an icon indicating an estimated subject from a display state to a non-display state, different transition methods are adopted in steps S8509 and S8510, so that CPU3 can notify the user that the estimated subject has changed. The transition methods in steps S8509 and S8510 may be different from each other. For example, contrary to the example of FIG. 8E, the transition method in step S8509 may be a method of gradually changing the size, and the transition method in step S8510 may be a method of gradually changing the transparency.

[0137] Step S8511 is a case where only one subject is detected and it is determined that the "icon should be hidden." CPU 3 prioritizes responsiveness and immediately transitions the icon indicating the estimated subject to a hidden state.

[0138] According to the above-mentioned embodiment 1, the camera 1 controls whether to display an icon indicating a subject (estimated subject) that is estimated to be the user's gaze direction, based on at least one of the image capture situation and the situation of the subject included in the image. The camera 1 can ensure visibility according to the situation and improve operability at the same time by automatically displaying or hiding the icon indicating the estimated subject.

[0139] Furthermore, the camera 1 transitions the icon indicating the estimated subject from one of a display state and a non-display state to the other in a transition method selected from a plurality of transition methods. For example, the camera 1 transitions the icon from one of a display state and a non-display state to the other by gradually changing the transparency or size of the icon. The gradual change in the icon state allows the user to concentrate on shooting without being unintentionally distracted. Furthermore, the gradual change in the icon state makes it easier for the user to recognize the position of the icon.

[0140] The icon display control process described in Fig. 8A shows an example of a process in which icon display determination based on the imaging situation (Fig. 8B) is followed by icon display determination based on the subject situation (Fig. 8C), but the order of determination is not limited to this. The CPU 3 may determine whether or not to display an icon indicating an estimated subject by determining at least one of the various conditions described as the imaging situation and the various conditions described as the subject situation in any order.

[0141] [Embodiment 2] The second embodiment will be described with reference to Figures 17A, 17B, 18, and 19. In the second embodiment, when switching between a display state and a non-display state of an icon indicating a subject at which a user's gaze is directed (a subject present at a position based on the gaze position, an estimated subject), a transition method is selected based on the state of the subject.

[0142] 8A to 8C are the same as those in embodiment 1. In embodiment 2, the processes shown in Figures 17A and 17B are executed instead of the processes shown in Figures 8D and 8E. The same processes as those in Figures 8D and 8E are denoted by the same reference numerals and detailed description thereof will be omitted.

[0143] 17A is a flowchart of the icon display process according to embodiment 2. If it is determined in step S8402 that the subject is being watched, the process proceeds to step S9001. If it is not determined in step S8404 that the estimated subject is in a "focused state," the process proceeds to step S9004.

[0144] In step S9001, the icon indicating the estimated subject is not displayed, and the user is paying attention to the estimated subject. The CPU 3 determines whether or not the distance between the estimated subject (the subject present at the position based on the gaze position) and the other subject is smaller than a predetermined threshold. When multiple subjects exist in the image, the other subject can be, for example, the subject closest to the estimated subject. The distance between the estimated subject and the other subject can be calculated, for example, using the coordinates of each subject in the image (for example, the coordinates of the center of gravity of each subject). When the distance between the estimated subject and the other subject is smaller than the predetermined threshold and the multiple subjects are close to each other, the process proceeds to step S9002. When the distance between the estimated subject and the other subject is larger than the predetermined threshold and the multiple subjects are not close to each other, the process proceeds to step S9003. Note that when the distance between the estimated subject and the other subject is the predetermined threshold, the CPU 3 may proceed to either step S9002 or step S9003.

[0145] In step S9002, where the multiple subjects are close to each other, the CPU 3 gradually changes (increases) the size of the icon representing the estimated subject, thereby transitioning the icon to a display state. On the other hand, in step S9003, where the multiple subjects are not close to each other, the CPU 3 gradually changes (decreases) the transparency of the icon representing the estimated subject, thereby transitioning the icon to a display state.

[0146] 18(a) to 18(d) are used to explain icon display control when an estimated subject and another subject are close to each other and the distance between the subjects is smaller than a predetermined threshold. FIGS. 18(a) to 18(d) are diagrams for explaining icon display for multiple subjects that are close to each other. In the example of FIG. 18(a), CPU 3 estimates, as the estimated subject, subject 1802, which is closer to the head of the detection target from gaze position 1803 (not displayed on display device 10) out of subjects 1801 and 1802.

[0147] In FIG. 18(a), subject 1801 and subject 1802 are close to each other on the image plane. In FIG. 18(b), CPU 3 starts a display process of icon 1804 indicating the estimated subject for subject 1802, who is the estimated subject (step S9002). In FIG. 18(b), icon 1804 is displayed in a size smaller than the head area of ​​estimated subject 1802. CPU 3 can notify the user of the position of the estimated subject by gradually increasing the size of icon 1804 from the state in FIG. 18(b) to FIG. 18(c) and then FIG. 18(d). Note that in FIGS. 18(b) to 18(d), gaze position 1803 is not displayed on display device 10.

[0148] 19(a) to 19(d) are used to explain icon display control in the case where an estimated subject is not close to another subject and the distance between the subjects is equal to or greater than a predetermined threshold. FIGS. 19(a) to 19(d) are diagrams for explaining icon display for multiple subjects that are not close to each other. In the example of FIG. 19(a), CPU 3 estimates, as the estimated subject, subject 1902, which is closer to the head to be detected from gaze position 1903 (not displayed on display device 10) out of subjects 1901 and 1902.

[0149] In FIG. 19(a), subject 1901 and subject 1902 are not close to each other on the image plane. In FIG. 19(b), CPU 3 starts a display process of icon 1904 indicating the estimated subject for subject 1902, which is the estimated subject (step S9003). In FIG. 19(b), icon 1904 is displayed with a density with a higher transmittance than normal display. CPU 3 gradually decreases the transmittance of icon 1904 from the state in FIG. 19(b) to FIG. 19(c) and then FIG. 19(d), thereby increasing the display density of icon 1904. This allows CPU 3 to notify the user of the position of the estimated subject. Note that in FIGS. 19(b) to 19(d), gaze position 1903 is not displayed on display device 10.

[0150] In step S9004 of Fig. 17A, the icon indicating the estimated subject is not displayed, and the estimated subject is not determined to be in focus, that is, the estimated subject is out of focus. Similar to step S9001, CPU 3 determines whether the distance between the estimated subject and the other subject is smaller than a predetermined threshold. If the distance between the estimated subject and the other subject is smaller than the predetermined threshold, the process proceeds to step S9005. If the distance between the estimated subject and the other subject is equal to or greater than the predetermined threshold, the process proceeds to step S9006.

[0151] In step S9005, where a plurality of subjects are close to each other, similarly to step S9002, CPU 3 gradually changes (increases) the size of the icon representing the estimated subject, thereby transitioning the icon to a display state. In step S9006, which is a situation where the subject is not very close, similarly to step S9003, CPU 3 gradually changes (reduces) the transparency of the icon indicating the estimated subject, thereby transitioning the icon to a display state.

[0152] According to the icon display control described in FIG. 17A, when the estimated subject at which the user's gaze is directed is close to other subjects, the CPU 3 can make the estimated subject easier to recognize by gradually increasing the size of the icon representing the estimated subject.

[0153] 17B is a flowchart of icon non-display processing according to embodiment 2. When a user's blink is detected and blink characteristic information is acquired in step S8503, the process proceeds to step S9011. When it is determined in step S8505 that the composition has been changed, the process proceeds to step S9014.

[0154] In step S9011, the icon representing the estimated subject is in a display state, and the icon is switched to a hidden state when the user blinks. CPU 3 determines whether or not the distance between the estimated subject and the other subject is smaller than a predetermined threshold, similar to step S9001 in FIG. 17A. If the distance between the estimated subject and the other subject is smaller than the predetermined threshold, the process proceeds to step S9012. If the distance between the estimated subject and the other subject is equal to or greater than the predetermined threshold, the process proceeds to step S9013.

[0155] In step S9012, where the multiple subjects are close to each other, the CPU 3 gradually changes (reduces) the size of the icon representing the estimated subject over a predetermined transition time, thereby transitioning the icon to a non-display state. On the other hand, in step S9013, where the multiple subjects are not close to each other, the CPU 3 gradually changes (increases) the transparency of the icon representing the estimated subject over a predetermined transition time, thereby transitioning the icon to a non-display state. The predetermined transition time is determined in the same manner as in step S8504 of FIG. 8E.

[0156] In step S9014, the icon indicating the estimated subject is in a display state, and after the composition is changed, a predetermined time has passed and the icon is switched to a non-display state. CPU 3 determines whether or not the distance between the estimated subject and the other subject is smaller than a predetermined threshold, as in step S9001 in FIG. 17A. If the distance between the estimated subject and the other subject is smaller than the predetermined threshold, the process proceeds to step S9015. If the distance between the estimated subject and the other subject is equal to or greater than the predetermined threshold, the process proceeds to step S9016.

[0157] In step S9015, where the multiple subjects are close to each other, the CPU 3 gradually changes (reduces) the size of the icon representing the estimated subject, thereby transitioning the icon to a non-display state. On the other hand, in step S9016, where the multiple subjects are not close to each other, the CPU 3 gradually changes (increases) the transparency of the icon representing the estimated subject, thereby transitioning the icon to a non-display state.

[0158] According to the above-mentioned embodiment 2, the camera 1 can improve the visibility of the icon and the image by appropriately selecting a transition method between a display state and a non-display state of the icon indicating the estimated subject based on the state of the subject. By appropriately selecting the transition method, the user can easily recognize the icon indicating the estimated subject.

[0159] In addition, in FIGS. 17A and 17B, the subject situation is described as the distance between the estimated subject and other subjects, but is not limited to this. The subject situation may be the number of subjects, the size of the subject, the brightness of the subject (luminance information), etc. The camera 1 appropriately selects a transition method between the display state and the non-display state according to these subject situations, thereby making it possible to display icons and images. The visibility of the image can be improved.

[0160] [Embodiment 3] A third embodiment will be described with reference to Figures 20A and 20B. In the second embodiment, when switching between a display state and a non-display state of an icon indicating a subject (estimated subject) at which the user's gaze is directed, a transition method is selected based on the state of the subject. In contrast, in the third embodiment, the transition method when switching between a display state and a non-display state of an icon indicating an estimated subject is selected based on the zoom state at the time of shooting.

[0161] 8A to 8C are the same as those in embodiment 1. In embodiment 3, the processes shown in Figures 20A and 20B are executed instead of the processes shown in Figures 8D and 8E. The same processes as those in Figures 8D and 8E are denoted by the same reference numerals and detailed description thereof will be omitted.

[0162] 20A is a flowchart of the icon display process according to embodiment 3. If it is determined in step S8402 that the subject is being watched, the process proceeds to step S9021. If it is not determined in step S8404 that the estimated subject is in a "focused state," the process proceeds to step S9024.

[0163] In step S9021, the icon indicating the estimated subject is not displayed, and the user is paying attention to the estimated subject. CPU 3 acquires information on the current zoom state and determines whether or not shooting is on the wide-angle side. CPU 3 can acquire information on whether or not the current zoom state (zoom position) is on the wide-angle side from zoom drive section 120 of photographing lens unit 1A that controls the zoom state. Also, when the angle of view has been changed by electronic zoom, CPU 3 can determine whether or not shooting is on the wide-angle side based on the setting state of electronic zoom on the camera side. If shooting is on the wide-angle side, processing proceeds to step S9022. If shooting is not on the wide-angle side, processing proceeds to step S9023.

[0164] In step S9022, which is shooting on the wide-angle side, CPU 3 transitions the icon to a display state by gradually changing (reducing) the transparency of the icon indicating the estimated subject. On the other hand, in step S9023, which is not shooting on the wide-angle side, CPU 3 transitions the icon to a display state by gradually changing (increasing) the size of the icon indicating the estimated subject. Generally, when shooting on the telephoto zoom side, the subject is viewed as crowded due to a compression effect. For this reason, when shooting on the telephoto side (not wide-angle side), CPU 3 gradually increases the size of the icon, making it easier to recognize the estimated subject and improving the visibility of the icon and image.

[0165] In step S9024, the icon indicating the estimated subject is not displayed, and the estimated subject is not determined to be in focus, that is, the estimated subject is out of focus. CPU 3 determines whether or not the image is shot on the wide-angle side, as in step S9021. If the image is shot on the wide-angle side, the process proceeds to step S9025. If the image is not shot on the wide-angle side, the process proceeds to step S9026.

[0166] In step S9025, which is wide-angle shooting, similar to step S9022, CPU 3 transitions the icon to a display state by gradually changing (lowering) the transparency of the icon indicating the estimated subject. On the other hand, in step S9026, which is telephoto shooting, CPU 3 transitions the icon to a display state by gradually changing (increasing) the size of the icon indicating the estimated subject, similar to step S9023.

[0167] According to the icon display control described in FIG. 20A, when the subject is viewed in a crowded state due to the compression effect in telephoto shooting, the CPU 3 reduces the size of the icon indicating the estimated subject. By gradually increasing the size of the image, the estimated subject can be easily recognized.

[0168] 20B is a flowchart of icon non-display processing according to embodiment 3. When a user's blink is detected and blink characteristic information is acquired in step S8503, the process proceeds to step S9031. When it is determined in step S8505 that the composition has been changed, the process proceeds to step S9034.

[0169] In step S9031, the icon representing the estimated subject is in a display state, and the user blinks, causing the icon to be switched to hidden. CPU 3 determines whether or not shooting is on the wide-angle side, similarly to step S9021 in FIG. 20A. If shooting is on the wide-angle side, the process proceeds to step S9032. If shooting is not on the wide-angle side (shooting is on the telephoto side), the process proceeds to step S9033.

[0170] In step S9032 when shooting on the wide-angle side, CPU 3 gradually changes (increases) the transparency of the icon indicating the estimated subject over a predetermined transition time, thereby transitioning the icon to a non-display state. On the other hand, in step S9033 when shooting on the telephoto side, CPU 3 gradually changes (reduces) the size of the icon indicating the estimated subject over a predetermined transition time, thereby transitioning the icon to a non-display state. The predetermined transition time is determined in the same manner as in step S8504. In the case of shooting on the telephoto side (not shooting on the wide-angle side), CPU 3 gradually reduces the size of the icon, thereby making it easier to recognize the estimated subject and improving the visibility of the icon and image.

[0171] In step S9034, the icon representing the estimated subject is in a display state, and after the composition is changed, a predetermined time has passed and the icon is switched to non-display. CPU 3 determines whether or not the image is shot on the wide-angle side, as in step S9031. If the image is shot on the wide-angle side, the process proceeds to step S9035. If the image is not shot on the wide-angle side, the process proceeds to step S9036.

[0172] In step S9035 when shooting on the wide-angle side, CPU 3 gradually changes (increases) the transparency of the icon indicating the estimated subject, thereby transitioning the icon to a non-display state. On the other hand, in step S9036 when shooting on the telephoto side, CPU 3 gradually changes (reduces) the size of the icon indicating the estimated subject, thereby transitioning the icon to a non-display state.

[0173] According to the above-mentioned embodiment 3, the camera 1 can improve the visibility of the icon and the image by appropriately selecting a transition method between a display state and a non-display state of the icon indicating the estimated subject based on the zoom state information at the time of shooting. By appropriately selecting the transition method, the user can easily recognize the icon indicating the estimated subject.

[0174] In the above-mentioned embodiments, an example has been described in which a transition method for transitioning an icon indicating an estimated subject from one of a display state and a non-display state to the other based on various conditions. However, the camera 1 may select a transition method based on conditions other than the above. For example, the camera 1 may select a transition method based on the movement of the subject (information on the movement characteristics), such as whether the subject present at a position based on the user's gaze position is stationary or moving while accelerating. For example, when a subject moving while accelerating stops, the camera 1 transitions the icon to a non-display state by gradually changing (increasing) the transparency of the icon indicating the estimated subject. The camera 1 may also select a transition method based on whether detection of the user's gaze position has succeeded or failed. Furthermore, the camera 1 may select a transition method based on a condition that is an arbitrary combination of these conditions. The transition method to be selected based on various conditions may be determined in consideration of the visibility of the icon and the screen. You can set it in advance.

[0175] The above-described embodiment is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0176] For example, although an example of viewing a subject through an EVF (electronic viewfinder) has been described, the present invention is also applicable to viewing a subject through an OVF (optical viewfinder). The present invention is applicable to devices other than imaging devices (cameras), and is also applicable to viewing three-dimensional computer graphics, etc. The present invention is applicable to various electronic devices capable of detecting the viewpoint.

[0177] [Applications to other electronic devices] Fig. 21(a) is an external view of a notebook personal computer 2110 (notebook PC). In Fig. 21(a), an imaging unit 2115 that captures an image of a user looking at a display unit 2111 of the notebook PC 2110 is connected to the notebook PC 2110, and the notebook PC 2110 obtains the imaging result from the imaging unit 2115. Then, the notebook PC 2110 detects the user's viewpoint based on the imaging result. The present invention is also applicable to the notebook PC 2110.

[0178] Fig. 21(b) is an external view of a smartphone 2120. In Fig. 21(b), the smartphone 2120 detects the viewpoint of a user looking at a display unit 2122 of the smartphone 2120 based on an imaging result of an in-camera 2121 (front camera). The present invention is also applicable to the smartphone 2120. Similarly, the present invention is also applicable to various tablet terminals.

[0179] FIG. 21(c) is an external view of the game machine 2130. In FIG. 21(c), a head mounted display 2135 (HMD) that displays a VR (Virtual Reality) image of a game on a display unit 2136 is connected to the game machine 2130. The HMD 2135 has a camera 2137 that captures an image of the eye of a user wearing the HMD 2135, and the game machine 2130 acquires the image capture result from the HMD 2135. Then, the game machine 2130 detects the user's viewpoint based on the image capture result. The present invention is also applicable to the game machine 2130. The components of the game machine 2130 may be built into the HMD 2135. Therefore, the present invention is also applicable to the HMD 2135. Just as the present invention is applicable to a case where a VR image displayed on an HMD is viewed, the present invention is also applicable to a case where an AR (Augmented Reality) image displayed on the lens part of a glasses-type wearable terminal or an actual object beyond the lens part is viewed. Just as the present invention is applicable to VR technology and AR technology, the present invention is also applicable to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

[0180] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0181] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) An acquisition means for acquiring information on a gaze position of a user viewing a captured image; a control means for controlling to display an item at a position based on the user's line of sight; having The control means transitions the item from one of a display state and a non-display state to the other in a transition manner selected from a plurality of transition manners. 1. An electronic device comprising: (Configuration 2) The plurality of transition methods include: a transition scheme in which the item transitions from one of a visible state and a hidden state to the other by gradually changing the size of the item; a transition scheme in which the item is transitioned from one of a visible state and a hidden state to the other by gradually changing the transparency of the item; a transition method for immediately transitioning the item from one of a displayed state and a hidden state to the other. 2. The electronic device according to configuration 1. (Configuration 3) The method further includes estimating an object at which the user's gaze is directed, among objects included in the image, based on the user's gaze position, The control means controls so as to display the item at the position of the subject estimated by the estimation means. 3. The electronic device according to configuration 1 or 2. (Configuration 4) The control means selects a transition method for transitioning the item to a display state based on whether the user is paying attention to the image. 4. The electronic device according to any one of configurations 1 to 3. (Configuration 5) The control means selects a transition method for transitioning the item to a hidden state based on whether or not the user blinks. 5. The electronic device according to any one of configurations 1 to 4. (Configuration 6) When the user blinks, the control means gradually changes the state of the item over a period of time longer than a previously measured period of time during which the user's eyes are closed while blinking, thereby transitioning the item from a displayed state to a hidden state. 6. The electronic device according to configuration 5. (Configuration 7) The control means selects a transition method for transitioning the item to a hidden state based on whether a composition of the image has been changed by a zoom operation, a pan operation, or a tilt operation. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 8) The control means selects a transition method for transitioning the item to a hidden state based on whether or not a composition of the image has been changed due to a change in the attitude of an imaging device that captures the image. 8. The electronic device according to any one of configurations 1 to 7. (Configuration 9) The control means selects a transition method for transitioning the item to a display state based on whether a subject present at a position based on the user's line of sight is in a focused state or a non-focused state. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 10) The control means selects a transition method for transitioning the item to a hidden state based on whether or not the image includes multiple subjects. 10. The electronic device according to any one of configurations 1 to 9. (Configuration 11) The control means causes the item to transition to a hidden state by a different transition method depending on whether the image includes a plurality of subjects and the subject currently viewed by the user is the same as or different from a subject previously viewed by the user. 11. The electronic device according to any one of configurations 1 to 10. (Configuration 12) The subject included in the image is a subject of a predetermined type. 12. The electronic device according to configuration 10 or 11. (Configuration 13) The number of subjects included in the image is the number of subjects included in a predetermined range including the gaze position of the user. 13. The electronic device according to any one of configurations 10 to 12. (Configuration 14) The predetermined range is determined based on at least one of the focal length, shooting distance, shooting mode, and the size of a subject present at a position based on the user's line of sight when capturing the image. 14. The electronic device according to configuration 13. (Configuration 15) The control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other, based on a distance between a subject present at a position based on the user's gaze position and another subject. 15. The electronic device according to any one of configurations 1 to 14. (Configuration 16) The control means transitions the item from one of a displayed state and a hidden state to the other by a different transition method depending on whether a distance between the subject present at a position based on the user's gaze position and another subject is smaller than a predetermined threshold or larger than the predetermined threshold. 16. The electronic device according to configuration 15. (Configuration 17) The control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other, based on a zoom state when the image is captured. 17. The electronic device according to any one of configurations 1 to 16. (Configuration 18) The control means transitions the item from one of a display state and a non-display state to the other in a different transition manner depending on whether the zoom state is on the wide-angle side or the telephoto side. 18. The electronic device according to configuration 17. (Configuration 19) The control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other, based on a movement of a subject that is present at a position based on the user's gaze position. 19. The electronic device according to any one of configurations 1 to 18. (Configuration 20) The control means transitions the item from one of the display state and the non-display state to the other in a different transition manner depending on whether detection of the user's gaze position is successful or unsuccessful. 20. The electronic device according to any one of configurations 1 to 19. (method) An acquisition step of acquiring information on a gaze position of a user viewing a captured image; a control step of controlling to display the item at a position based on the user's gaze position; having In the control step, the item is transitioned from one of a display state and a non-display state to the other in a transition manner selected from a plurality of transition manners. 23. A method for controlling an electronic device comprising: (program) A program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 20. (medium) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 20. [Explanation of symbols]

[0182] 1: camera (electronic device), 3: CPU, 10: display device, 11: display device drive circuit, 17: eye image sensor, 201: line of sight detection circuit, 208: image processing unit

Claims

1. An acquisition means for acquiring information on a gaze position of a user viewing a captured image; a control means for controlling the display of an item at a position based on the user's line of sight; The control means transitions the item from one of a display state and a non-display state to the other in a transition manner selected from a plurality of transition manners.

1. An electronic device comprising:

2. The plurality of transition methods include: a transition scheme in which the item transitions from one of a visible state and a hidden state to the other by gradually changing the size of the item; a transition scheme in which the item is transitioned from one of a visible state and a hidden state to the other by gradually changing the transparency of the item; a transition method for immediately transitioning the item from one of a displayed state and a hidden state to the other.

2. The electronic device according to claim 1 .

3. The method further includes estimating an object at which the user's gaze is directed, among objects included in the image, based on the user's gaze position, The control means controls so as to display the item at the position of the subject estimated by the estimation means.

2. The electronic device according to claim 1 .

4. The control means selects a transition method for transitioning the item to a display state based on whether the user is paying attention to the image.

2. The electronic device according to claim 1 .

5. The control means selects a transition method for transitioning the item to a hidden state based on whether or not the user blinks.

2. The electronic device according to claim 1 .

6. When the user blinks, the control means gradually changes the state of the item over a period of time longer than a previously measured period of time during which the user's eyes are closed while blinking, thereby transitioning the item from a displayed state to a hidden state.

6. The electronic device according to claim 5,

7. The control means selects a transition method for transitioning the item to a hidden state based on whether a composition of the image has been changed by a zoom operation, a pan operation, or a tilt operation.

2. The electronic device according to claim 1 .

8. The control means selects a transition method for transitioning the item to a hidden state based on whether or not a composition of the image has been changed due to a change in the attitude of an imaging device that captures the image.

2. The electronic device according to claim 1 .

9. The control means selects a transition method for transitioning the item to a display state based on whether a subject present at a position based on the user's line of sight is in a focused state or a non-focused state.

2. The electronic device according to claim 1 .

10. The control means selects a transition method for transitioning the item to a hidden state based on whether or not the image includes multiple subjects.

2. The electronic device according to claim 1 .

11. The control means causes the item to transition to a hidden state by a different transition method depending on whether the image includes a plurality of subjects and the subject currently viewed by the user is the same as or different from a subject previously viewed by the user.

2. The electronic device according to claim 1 .

12. The subject included in the image is a subject of a predetermined type.

11. The electronic device according to claim 10.

13. The number of subjects included in the image is the number of subjects included in a predetermined range including the gaze position of the user.

11. The electronic device according to claim 10.

14. The predetermined range is determined based on at least one of the focal length, shooting distance, shooting mode, and the size of a subject present at a position based on the user's line of sight when capturing the image.

14. The electronic device according to claim 13.

15. The control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other, based on a distance between a subject present at a position based on the user's gaze position and another subject.

2. The electronic device according to claim 1 .

16. The control means transitions the item from one of a displayed state and a hidden state to the other by a different transition method depending on whether a distance between the subject present at a position based on the user's gaze position and another subject is smaller than a predetermined threshold or larger than the predetermined threshold.

16. The electronic device according to claim 15.

17. The control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other, based on a zoom state when the image is captured.

2. The electronic device according to claim 1 .

18. The control means transitions the item from one of a display state and a non-display state to the other in a different transition manner depending on whether the zoom state is on the wide-angle side or on the telephoto side.

18. The electronic device according to claim 17 .

19. The control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other, based on a movement of a subject that is present at a position based on the user's gaze position.

2. The electronic device according to claim 1 .

20. The control means transitions the item from one of the display state and the non-display state to the other in a different transition manner depending on whether detection of the user's gaze position is successful or unsuccessful.

2. The electronic device according to claim 1 .

21. An acquisition step of acquiring information on a gaze position of a user viewing a captured image; a control step of controlling to display the item at a position based on the user's gaze position; having In the control step, the item is transitioned from one of a display state and a non-display state to the other in a transition manner selected from a plurality of transition manners.

23. A method for controlling an electronic device comprising:

22. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 20.

23. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 20.

Citation Information

Patent Citations

  • Camera provided with line-of-sight detecting function

    JP1996184746A

  • Electronic apparatus

    JP2021131441A