Electronic device, control method thereof, and computer-readable storage medium

Abstract

An electronic device includes: a display control unit configured to display an input image and a first GUI image at a display unit; a first detection unit configured to detect a user's viewpoint in the input image; and a second detection unit configured to detect a position of a predetermined object in the input image, and when the position of the predetermined object is detected based on the viewpoint, the display control unit makes the first GUI image less prominent than when the position of the predetermined object is not detected based on the viewpoint.

Description

Technical Field

[0001] The present invention relates to an electronic device, a method for controlling an electronic device, and a computer-readable storage medium. Background Art

[0002] In recent years, cameras have become more automated and intelligent, and technologies have been proposed for detecting an object intended by a user and performing focus control based on information about the line of sight position of the user looking through a viewfinder (line of sight information; line of sight input information) without manually inputting the position of the object. PTL 1 discloses a technology for improving the accuracy of object detection by correcting an area detected as an object area using line of sight information.

[0003] [Citation List]

[0004] [Patent Document]

[0005] [PTL 1] Japanese Patent Application No. 2018-205648 Summary of the invention

[0006] [Technical issues]

[0007] When a GUI (Graphical User Interface) image is displayed in an area other than the area desired by the user (the area that the user wants to see) when obtaining the line of sight position of the user, the line of sight of the user tends to be directed to the displayed GUI image, and it is difficult to keep the line of sight in the desired area. Therefore, even if the line of sight information is used, the position of the area detected as the object area and the actual position of the object may be offset from each other.

[0008] Therefore, the present invention allows the user's sight to be directed to a desired area more easily.

[0009] [Solutions to solve the problem]

[0010] One aspect of the present invention is:

[0011] An electronic device, comprising:

[0012] a display control unit configured to display the input image and the first GUI image at the display unit;

[0013] A first detection unit configured to detect a viewpoint of a user in the input image; and

[0014] A second detection unit is configured to detect a position of a predetermined object in the input image, wherein:

[0015] The display control unit makes the first GUI image less noticeable in a case where the position of the predetermined object is detected based on the viewpoint, compared to a case where the position of the predetermined object is not detected based on the viewpoint.

[0016] [Advantageous Effects of the Invention]

[0017] According to the present invention, the user's sight can be directed to a desired area more easily. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] [ Figure 1 ]

[0019] Figure 1 is a cross-sectional view of a camera according to an embodiment of the present invention.

[0020] [ Figure 2 ]

[0021] Figure 2 is a block diagram of a camera according to an embodiment.

[0022] [ Figure 3 ]

[0023] Figure 3 (A) to (C) are views for illustrating the field of view in the viewfinder according to the embodiment.

[0024] [ Figure 4 ]

[0025] Figure 4 is a view for illustrating the principle of a method for detecting a field of view according to an embodiment.

[0026] [ Figure 5 ]

[0027] Figure 5 (A) and (B) show eye images according to an embodiment.

[0028] [ Figure 6 ]

[0029] Figure 6 is a flowchart for illustrating a line of sight detection operation according to an embodiment.

[0030] [ Figure 7 ]

[0031] Figure 7 (A) to (D) show examples of display of a conventional live view image.

[0032] [ Figure 8 ]

[0033] Figure 8is a flowchart for illustrating tracking processing according to the first embodiment of the present invention.

[0034] [ Fig. 9 ]

[0035] Fig. 9 (A) to (D) show examples of display of a live view image according to the first embodiment.

[0036] [ Fig.10 ]

[0037] Fig.10 is a flowchart illustrating a tracking process according to the second embodiment of the present invention.

[0038] [ Fig.11 ]

[0039] Fig.11 (A) to (D) show examples of display of a live view image according to the second embodiment.

[0040] [ Fig.12 ]

[0041] Fig.12 is a block diagram of a camera according to a third embodiment of the present invention.

[0042] [ Fig.13 ]

[0043] Fig.13 is a flowchart for illustrating face detection processing according to the third embodiment. DETAILED DESCRIPTION

[0044] Preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0045] (First embodiment)

[0046] <Configuration Description>

[0047] Figure 1 is a cross-sectional view of a camera 1 (digital still camera; interchangeable lens camera) according to the embodiment, and shows a general internal configuration of the camera 1. The camera 1 includes a photographic lens unit 1A and a camera housing 1B.

[0048] The photographic lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mounting contact 117, and a focus circuit 118. The lens drive member 114 includes a drive gear, and the photocoupler 115 detects the rotation of the pulse plate 116 linked to the lens drive member 114, and sends the detection to the focus circuit 118. The focus circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (information about the lens drive amount), and changes the focus position by moving the lens 101. The mounting contact 117 is an interface between the photographic lens unit 1A and the camera housing 1B. For simplicity, two lenses 101 and 102 are shown, but in reality, more than two lenses are included in the photographic lens unit 1A.

[0049] The camera housing 1B stores an image sensing device 2, a CPU 3, a memory unit 4, a display device 10, a display device driving circuit 11, and an eyepiece 12. The image sensing device 2 is provided at the surface of the photographic lens unit 1A to form an image thereon. The CPU 3 is a central processing unit of a microcomputer, and controls the entire camera 1. The memory unit 4 stores, for example, an image captured by the image sensing device 2. The display device 10 includes liquid crystal and other elements, and displays, for example, a captured image (subject image). The display device driving circuit 11 drives the display device 10. The eyepiece 12 is a lens for observing an image (an image for visual recognition) displayed on the display device 10.

[0050] The camera housing 1B also stores light sources 13a and 13b, a beam splitter 15, a light receiving lens 16, and an eye image sensing device 17. Each of the light sources 13a and 13b is a conventional light source used in a single-lens reflex camera to detect the direction of sight based on the relationship between a reflected image reflected by the cornea (cornea reflected image) and a pupil and illuminate the user's eyeball 14. Specifically, the light sources 13a and 13b may be infrared light emitting diodes that emit infrared light to which the user is not sensitive, and are arranged around the eyepiece lens 12. An optical image of the illuminated eyeball 14 (eyeball image; an image formed by light emitted from the light sources 13a and 13b and reflected by the eyeball 4) is transmitted through the eyepiece 12 and reflected by the beam splitter 15. Then, the image of the eyeball is formed by the light receiving lens 16 on the eye image sensing device 17, which has a two-dimensional arrangement of photoelectric conversion elements such as CCDs. The light receiving lens 16 positions the pupil of the eyeball 14 and the eye image sensing device 17 in a conjugate imaging relationship. Using a predetermined algorithm which will be described, a line of sight direction (viewpoint in the viewing image) is detected from the position of the cornea reflection image in the eyeball image formed on the eye image sensing device 17 .

[0051] Figure 2 1 is a block diagram of an electrical configuration in the camera 1. The CPU 3 is connected to a sight line detection circuit 201, a photometry circuit 202, an automatic focus detection circuit 203, a display device drive circuit 11, a light source drive circuit 205, and a tracking processing circuit 207. The CPU 3 transmits signals to a focus adjustment circuit 118 provided in the photographic lens unit 1A and an aperture control circuit 206 included in an aperture drive unit 112 in the photographic lens unit 1A through a mounting contact 117. The memory unit 4 associated with the CPU 3 has a function of storing image sensing signals from the image sensing device 2 and the eye image sensing device 17, and a function of storing sight line correction parameters for correcting individual differences in sight lines, which will be described.

[0052] While the eye image is formed on the eye image sensing device 17 (CCD-EYE), the sight line detection circuit 201 performs A / D conversion on the output of the eye image sensing device 17 (the eye image obtained by capturing the image of the eye), and sends the result to the CPU 3. The CPU 3 extracts feature points required for the sight line detection operation from the eye image according to a predetermined algorithm to be described, and calculates the user's sight line (the viewpoint in the image used for visual recognition) based on the position of the feature point.

[0053] The photometric circuit 202 amplifies, logarithmically compresses, or A / D converts a signal obtained from the image sensing device 2 (also used as a photometric sensor) (specifically, a brightness signal corresponding to the brightness of the subject field), and sends the result to the CPU 3 as subject field brightness information.

[0054] The automatic focus detection circuit 203 performs A / D conversion on the signal voltages from the plurality of detection elements (a plurality of pixels) included in the CCD in the image sensing device 2 and used for phase difference detection, and sends the result to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point based on the signals from the plurality of detection elements. This is a technique called imaging plane phase difference detection AF. According to this embodiment, as an example, it is assumed that the phase difference detection points on the image sensing plane are the same as those on the image sensing plane. Figure 3 There are focus detection points at 180 positions corresponding to the 180 positions shown in the field image (image for visual recognition) in the viewfinder in (A).

[0055] The tracking processing circuit 207 performs object tracking processing based on the image captured by the image sensing device 2. Specifically, the tracking processing circuit 207 performs matching processing between the current frame image (current frame image) and the reference image, and extracts (detects) the area with the highest similarity (correlation) with the reference image from the current frame image as the tracking target area. According to an embodiment, the reference image is an image (partial image) of an area detected as the tracking target area in the previous frame image (the frame image before the current frame image). The similarity can be the similarity between the feature quantities of the images. Then, the tracking processing circuit 207 determines whether to continue tracking based on the similarity. The tracking processing circuit 207 calculates the tracking reliability (reliability of the tracking processing; tracking state value) based on the similarity corresponding to the detected area, and sends the result to the CPU 3. The tracking processing is performed for, for example, automatic focus control, and the tracking reliability is used for, for example, display control of a GUI that displays a subject.

[0056] Figure 3 (A) shows the field of view in the viewfinder when the display device 10 is in an operating state (displaying an image for visual recognition). Figure 3 As shown in (A), the field of view in the viewfinder may include a focus detection area 300, 180 ranging indicators 301, and a field of view mask 302. Each of the 180 ranging indicators 301 is superimposed on a through image (live view image) displayed at the display device 10, so that these indicators are displayed at positions corresponding to focus detection points on the image sensing surface. Among the 180 ranging indicators 301, the ranging indicator 301 corresponding to the current viewpoint A (estimated position) is emphasized, for example, using a frame.

[0057] <Description of line of sight detection operation>

[0058] Will refer to Figure 4 , Figure 5 (A) Figure 5 (B) and Figure 6 Describe the gaze detection method. Figure 4 FIG. 1 shows the principle of the sight line detection method and is a schematic diagram of an optical system for sight line detection. Figure 4 As shown, light sources 13a and 13b are positioned approximately symmetrically with respect to the optical axis of light receiving lens 16 and illuminate the user's eyeball 14. A portion of light emitted from light sources 13a and 13b and reflected by eyeball 14 is captured by light receiving lens 16 and collected by eye image sensing device 17. Figure 5 (A) schematically shows an eye image captured by the eye image sensing device 17 (an eye image projected on the eye image sensing device 17), and Figure 5 (B) shows the output intensity of the CCD in the ocular image sensing device 17 . Figure 6is a flowchart schematically illustrating a line of sight detection operation.

[0059] When the line of sight detection operation starts, Figure 6 In step S601, the light sources 13a and 13b emit infrared light to the user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on the eye image sensing device 17 through the light receiving lens 16 and photoelectrically converted by the eye image sensing device 17. In this way, an electrical signal for a processable eye image can be obtained.

[0060] In step S602 , the visual line detection circuit 201 transmits the eye image (eye image signal; electric signal for the eye image) obtained from the eye image sensing device 17 to the CPU 3 .

[0061] In step S603, the CPU 3 obtains the coordinates of points corresponding to the cornea reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S602.

[0062] The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the cornea reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 142 are focused by the light receiving lens 16 and formed on the eye image sensing device 17 to become cornea reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the edges a and b of the pupil 141 are also focused on the eye image sensing device 17 to form pupil edge images a' and b' in the eye image.

[0063] Figure 5 (B) shows the Figure 5 Brightness information (brightness distribution) of region α′ in the eye image in (A). Figure 5 (B) shows the brightness distribution in the X-axis direction, where the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction. According to this embodiment, the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil edge images a' and b' in the X-axis direction (horizontal direction) are Xa and Xb. Figure 5As shown in (B), at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe', there are extremely high brightness levels. In the area from coordinates Xa to Xb corresponding to the area of ​​the pupil 141 (the area of ​​the pupil image obtained when the light beam from the pupil 141 forms an image on the eye image sensing device 17), there are extremely low levels of brightness except at coordinates Xd and Xe. Then, an intermediate brightness between the above two brightness levels is obtained in the area of ​​the iris 143 outside the pupil 141 (the area of ​​the iris image formed by the light beam from the iris 143 outside the pupil image). Specifically, the intermediate brightness between the above two levels of brightness is obtained in the area where the X coordinate (coordinate in the X-axis direction) is less than the coordinate Xa and in the area where the X coordinate is greater than the coordinate Xb.

[0064] from Figure 5 From the brightness distribution shown in (B), the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, coordinates with extremely high brightness levels can be obtained as coordinates of the corneal reflection images Pd' and Pe', and coordinates with extremely low brightness levels can be obtained as coordinates of the pupil edge images a' and b'. In the case where the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the pupil center image c' (the center of the pupil image) formed by the light beam from the pupil center c on the eye image sensing device 17 can be expressed as Xc≈(Xa+Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated based on the X-coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0065] In step S604, the CPU 3 calculates the image formation magnification β of the eyeball image. The image formation magnification β is a magnification determined by the position of the eyeball 14 relative to the light receiving lens 16 and can be obtained using a function of the distance (Xd-Xe) between the cornea reflection images Pd' and Pe'.

[0066] In step S605, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16. The X coordinate of the midpoint between the cornea reflection image Pd and the cornea reflection image Pe substantially coincides with the X coordinate of the center of curvature O of the cornea 142. Therefore, when the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, the rotation angle θx of the eyeball 14 in the ZX plane (a plane perpendicular to the Y axis) can be calculated using the following expression 1. The rotation angle θy of the eyeball 14 in the ZY plane (a plane perpendicular to the X axis) can also be calculated in the same manner as the method of calculating the rotation angle θx.

[0067] β×Oc×SINθ X ≈{(Xd+Xe) / 2}-Xc(Expression 1)

[0068] In step S606, the CPU 3 uses the rotation angles θx and θy calculated in step S605 to obtain (estimate) the viewpoint of the user (the position where the line of sight is focused; the position where the user is looking) in the image for visual recognition displayed on the display device 10. Assuming that the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated using the following expressions 2 and 3.

[0069] Hx=m×(Ax×θx+Bx) (Expression 2)

[0070] Hy=m×(Ay×θy+By) (Expression 3)

[0071] The parameter m in Expressions 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16 and other elements) of the camera 1. The parameter m is a conversion factor for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c in the image for visual recognition, and is predetermined and stored in the memory unit 4. The parameters Ax, Bx, Ay, and By are line of sight correction parameters for correcting individual differences in line of sight, and these parameters are obtained by performing a calibration operation to be described and are stored in the memory unit 4 before the line of sight detection operation is started.

[0072] In step S607 , the CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 4 , and ends the visual line detection operation.

[0073] The above description is about a method of obtaining the coordinates of the viewpoint (gaze point) on the display device by using the corneal reflection images of light sources 13a and 13b, however any other method for obtaining the coordinates of the viewpoint (eyeball rotation angle) from a captured eye image may be used.

[0074] <Description of calibration operation>

[0075] As described above, the viewpoint can be estimated by obtaining the rotation angles θx and θy of the eyeball 14 from the eye image in the line of sight detection operation and transforming the coordinates of the pupil center c to its position on the image for visual recognition.

[0076] However, due to factors such as individual differences in the shape of human eyes in some cases, the viewpoint cannot be estimated with high accuracy. Specifically, if the sight line correction parameters Ax, Ay, Bx, and By are not adjusted to values ​​suitable for the user, the viewpoint may be Figure 3As shown in (B), a gap is generated between the actual viewpoint B and the estimated viewpoint C. Figure 3 In (B), the user is looking at a person, but the camera 1 mistakenly estimates that the user's line of sight is looking at the background, and cannot perform appropriate focus detection and adjustment in this state.

[0077] Therefore, before the camera 1 captures an image, a calibration work must be performed to obtain viewpoint correction parameters suitable for the user and store these parameters in the camera 1 .

[0078] Traditionally, the image used for visual recognition is processed by analyzing the image before image sensing. Figure 3 The calibration work is performed by emphasizing multiple indicators at different positions shown in (C) and requiring the user to look at these indicators. It is known that the sight line detection operation is performed while the user's sight is looking at each indicator, and the viewpoint correction parameters suitable for the user are obtained based on multiple calculated viewpoints (estimated positions) and the coordinates of the indicators. If a position that the user should look at is suggested, it does not have to be indicated by an indicator, and the position can be highlighted by changing the brightness or color.

[0079] <Conventional Tracking Processing (Conventional Method)>

[0080] Figure 7 (A) to (D) show display examples in a frame image 701 by a conventional method. In the frame image 701, a subject 702 and a subject GUI 703 indicating an area for tracking the subject are displayed. The subject GUI 703 is a GUI image indicating an area (subject recognition area; recognition area) identified by the camera as a moving subject. The user indicates to the camera that the subject 702 is a subject to be tracked, for example, by touching the screen, and a video is obtained while capturing the subject on the screen.

[0081] Figure 7 (A) shows the state of the camera immediately after starting to track the object (subject) to be tracked, and the subject 702 and the subject GUI 703 coincide. At this time, the tracking reliability (for example, from 0% to 100%) indicating the degree of similarity between the subject 702 and the area surrounded by the subject GUI 703 is high. The tracking reliability can also be regarded as the reliability of the position of the subject GUI 703. Figure 7 (B) and (C) of FIG. 7 show how the subject GUI 703 gradually moves away from the subject 702 when the subject 702 moves at high speed or blends with background objects. Figure 7 In (D), when the tracking reliability is lower than the threshold value Th, the subject GUI 703 is not displayed and the tracking process stops.

[0082] As described above, when the subject moves at high speed or is mixed with background objects, it may not be possible to track the subject 702 through conventional tracking processing. Here, in the process for tracking the subject 702, the subject recognition area may be adjusted based on the user's line of sight 704. Figure 7 As shown in (D) of FIG. 7 , once the subject GUI 703 is no longer displayed, the line of sight 704 should be more likely to capture the subject 702, so it may be useful to adjust the subject recognition area based on the user's line of sight 704. Figure 7 As shown in (C), when the subject GUI 703 offset from the subject 702 is displayed, the line of sight 704 tends to be directed toward the subject GUI 703 and is unlikely to remain on the subject 702. Figure 7 If the subject recognition area is adjusted based on the line of sight 704 in the frame image shown in (C), a gap may be generated between the subject recognition area and the actual subject.

[0083] <Tracking Process Using Viewpoint (Example)>

[0084] Figure 8 1 is a flowchart for illustrating an exemplary tracking process according to the embodiment. The processing steps in the flowchart are performed when the CPU 3 executes a program stored in the memory unit 4. According to the embodiment, the processing in the flowchart starts when the power of the camera is turned on.

[0085] In step S801, when the power of the camera is turned on and the image sensing device 2 starts to obtain a through image (input image), the CPU 3 starts to display the through image on the display device 10. According to this embodiment, the user visually recognizes the subject by viewing the through image displayed on the display device 10 in the viewfinder.

[0086] In step S802, the CPU 3 determines whether the sight line input is turned on. According to this embodiment, when the sight line detection function (sight line input function) is turned on, the CPU 3 determines that the sight line input is turned on. If the sight line input is turned on, the process proceeds to step S803, and if not, the process proceeds to step S808.

[0087] In step S803, the CPU 3 executes Figure 6 According to this embodiment, when executing Figure 6 During the visual line detection routine shown, the coordinates (Hx, Hy) of the viewpoint on the display device 10 are obtained.

[0088] In step S804, the CPU 3 determines whether the subject GUI indicating the subject position and the line of sight GUI indicating the viewpoint (line of sight position; estimated gaze point) overlap. If they overlap, the process proceeds to step S806, and if they do not overlap, the process proceeds to step S805. For example, when a line as the subject GUI and a line as the line of sight GUI intersect or when one of the area of ​​the subject GUI and the area of ​​the line of sight GUI includes at least part of the other, the subject GUI and the line of sight GUI overlap. The subject GUI is displayed in the following step S813, and if the subject GUI is not displayed (for example, immediately after the start of the process), the process proceeds to step S805.

[0089] In step S805, the CPU 3 displays the line-of-sight GUI on the display device 10. According to this embodiment, the line-of-sight GUI is a frame surrounding the line-of-sight position of the user.

[0090] In step S806, the CPU 3 hides the sight line GUI. This is because if the subject GUI and the sight line GUI overlap on the subject in the display, the subject may not be clearly visible, which may cause discomfort to the user. It is only necessary to make the subject GUI easier to see, and the color of the sight line GUI may be lighter or the size may be smaller so that the sight line GUI may be less noticeable.

[0091] In step S807, CPU 3 sets threshold value Th to threshold value Th1. In step S808, CPU 3 sets threshold value Th to threshold value Th2 (<threshold value Th1). As will be described in detail, according to an embodiment, when the tracking reliability is equal to or less than threshold value Th, the subject GUI is erased (hidden) to prevent the line of sight from turning to the subject GUI that is offset from the subject. Therefore, threshold value Th1 for detecting the viewpoint is greater than threshold value Th2 for the case where the viewpoint is not detected. This makes it possible to quickly detect the start of the offset of the subject GUI when detecting the viewpoint, and to easily direct the line of sight to the subject. In the following description of the embodiment, threshold value Th1 is 70% and threshold value Th2 is 50%.

[0092] In step S809, the CPU 3 controls the tracking processing circuit 207 to perform moving object detection (tracking processing). It can be considered that the CPU 3 does not extract the position of the object based on the viewpoint. Various existing methods can be used as moving object detection methods. For example, as described above, matching processing is performed between frames, and the area with the highest similarity (correlation) between frames is detected as the subject recognition area. The CPU 3 also controls the tracking processing circuit 207 to calculate the tracking reliability based on the similarity corresponding to the detected area. The tracking reliability increases as the similarity increases.

[0093] In step S810, the CPU 3 sets the result in step S809 (the detection result of the region of the moving object) as the subject position information. The subject position information is used as, for example, a distance measurement point.

[0094] In step S811, the CPU 3 determines whether the tracking reliability is lower than the threshold value Th. If the tracking reliability is lower than the threshold value Th, the process proceeds to step S812, and if it is not lower than the threshold value Th, the process proceeds to step S813. Here, the case where the tracking reliability is lower than the threshold value Th is the case where the subject position and the subject GUI position are offset from each other. The case where the tracking reliability is higher than the threshold value Th is the case where the positions of the subject and the subject GUI are roughly coincident.

[0095] In step S812, the CPU 3 hides the subject GUI. It is only necessary to prevent the line of sight from being diverted to the subject GUI, and the subject GUI can be made less noticeable by making the subject GUI lighter in color or smaller in size. In addition to the subject GUI, an OSD (On Screen Display) that displays photography setting information on the photography screen may be displayed as a GUI in some cases. Because the line of sight may be moved away from the subject, the OSD may also interfere with the tracking process using the line of sight information, so it is preferable to change the display state of the OSD similarly to the subject GUI (to make the OSD less noticeable). If the subject GUI is hidden and only the line of sight GUI is displayed, it is also easier for the user to realize that the tracking process is being performed by the line of sight input.

[0096] In step S813, the CPU 3 displays the subject GUI. This is because there is a high possibility that the position of the subject and the position of the subject GUI roughly coincide with each other.

[0097] In step S814, CPU 3 updates the subject position information based on the line of sight information. Here, since the subject GUI is hidden in step S812, the possibility that the user is looking at the subject is high. Therefore, the subject recognition area is corrected by weighted addition of the line of sight position and the position of the subject recognition area to reduce the offset between the subject recognition area and the subject. Note that the area including the line of sight position (the area corresponding to the line of sight GUI) can be used as the subject recognition area as it is. When the reliability is lower than the threshold value Th, it can be considered that CPU 3 detects the position of the object that is not extracted based on the viewpoint and corrected based on the viewpoint in step S809 as the subject recognition area (the position detected based on the viewpoint). If the reliability is higher than the threshold value Th, it can be considered that CPU 3 detects the position of the extracted object (without correction based on the line of sight) as the subject recognition area (not the position detected based on the line of sight).

[0098] In step S815, the CPU 3 detects whether the user has turned on a photographing switch (not shown). If the switch has been turned on, the process proceeds to step S816, and if not, the process returns to step S802.

[0099] In step S816 , the CPU 3 performs a photographing operation. Specifically, the CPU 3 records the image signal obtained by the image sensing device 2 in the memory unit 4 .

[0100] In step S817, the CPU 3 determines whether the photography has been completed. If the photography has been completed, the processing flow ends, and if not, the processing returns to step S802. For example, when the power is turned off, the photography ends.

[0101] Fig. 9 (A) to (D) show exemplary frame images 701 in a moving image or a live view according to an embodiment.

[0102] exist Fig. 9 In (A), immediately after the subject 702 as a moving object starts to be tracked, the subject 702 and the subject GUI 703 overlap. At this time, the tracking reliability is high (for example, 90%). Fig. 9 In (A), an OSD 901 and a line-of-sight GUI 905 are also displayed. Here, since the line-of-sight detection function is turned on, the threshold Th is the threshold Th1 (for example, 70%).

[0103] exist Fig. 9 In (B), the position of the subject 702 begins to deviate from the subject GUI 703, but the tracking reliability (e.g., 80%) is higher than the threshold Th (threshold Th1). Fig. 9 In (B), the subject GUI 703 and the line of sight GUI 905 overlap each other. When the line of sight GUI 905 and the subject GUI 703 overlap, the information becomes complicated, which may cause discomfort to the user. When the subject GUI 703 and the line of sight GUI 905 overlap, the visibility of the subject GUI 703 is prioritized (in Fig. 9 In the example of (B), the line of sight GUI 905 is hidden).

[0104] exist Fig. 9 In (C), the positions of the subject 702 and the subject GUI 703 are greatly offset from each other, and the tracking reliability (for example, 60%) is less than the threshold Th (threshold Th1). Fig. 9 In the example of (C), the subject GUI 703 is hidden and less noticeable. Fig. 9In the example in (C), the color of OSD 901 becomes lighter to be less noticeable. According to this embodiment, when the sight line detection function is turned on, the start of the shift of the subject GUI 703 is quickly detected by using a relatively large threshold value Th1. When the start of the shift of the subject GUI 703 is detected, the subject GUI 703 is hidden or the color of OSD 901 becomes lighter, making it less likely that the user will view the subject GUI 703 or OSD 901, and the user's sight is more likely to be directed to the subject 702.

[0105] exist Fig. 9 In (D), as a result of the tracking process performed based on the line of sight position, the position of the subject GUI 703 and the position of the subject 702 roughly coincide with each other again. Fig. 9 In the example of (D), the tracking reliability (eg, 90%) is higher than the threshold Th (threshold Th1), the subject GUI 703 is displayed again, and the OSD 901 returns to its normal color. Fig. 9 In the example of (D), the line of sight GUI 905 is hidden because it overlaps with the subject GUI 703.

[0106] As described above, according to the embodiment, when the sight line detection function is turned on, the threshold value Th for evaluating the tracking reliability is increased. In this way, the start of the offset between the subject GUI and the subject position is quickly detected, and the subject GUI 703 is made to look less obvious, making it easier for the user's sight to be directed to the subject, which improves the accuracy of the tracking process using the sight line information. In addition, when the subject GUI is less obvious, the user is made aware that the tracking process is being performed using the sight line information, so that the user can more easily look at the subject.

[0107] (Second embodiment)

[0108] In the description of the first embodiment, the line-of-sight GUI is always displayed except when the position of the subject GUI and the line-of-sight position overlap, but according to this embodiment, the line-of-sight GUI is exemplified as being displayed only when tracking processing is performed using line-of-sight information.

[0109] Fig.10 is a flowchart showing an exemplary tracking process according to the embodiment. When the CPU 3 executes a program stored in the memory unit 4, the processing steps in the flowchart are performed. According to the embodiment, the processing in the flowchart starts when the power of the camera is turned on. Figure 8 The same processing steps shown are denoted by the same reference numerals, and a description thereof will not be provided.

[0110] According to the embodiment, no Figure 8Therefore, after the line of sight detection routine is performed, the process of displaying or hiding the line of sight GUI is not performed immediately. When the tracking reliability drops below the threshold value Th in step S811 and the subject GUI 703 is hidden, the line of sight GUI is displayed (step S1018). When the tracking reliability exceeds the threshold value Th in step S811 and the subject GUI 703 is displayed, the line of sight GUI is hidden (step S1019).

[0111] In this way, the user can be made aware that the gaze information is being used for tracking processing without displaying more gaze GUI than is necessary and causing discomfort to the user.

[0112] Fig.11 (A) to (D) each show a frame image 701 in a moving image or a live view according to an embodiment.

[0113] Fig.11 (A) shows the state immediately after the tracking of the subject 702 as a moving object is started, and the subject 702 and the subject GUI 703 are roughly overlapped (tracking reliability is higher than the threshold value Th1). In this case, if the line of sight GUI 905 and the subject GUI 703 are displayed on the display device 10 at the same time, the information on the display device 10 becomes complicated and may cause discomfort to the user, so unlike the first embodiment, the line of sight GUI 905 is not displayed.

[0114] Fig.11 (C) shows a state where the tracking reliability is less than the threshold value Th (Yes in step S811). In this case, the line of sight GUI 905 is displayed to make the user aware that the line of sight information is being used for tracking processing. Fig.11 In (B) of 11 and (D) of 11, since the tracking reliability is higher than the threshold value Th (No in step S811), it is similar to Fig.11 (A), the sight line GUI 905 is hidden.

[0115] As described above, according to the embodiment, the line of sight GUI is displayed only when the line of sight information is used for tracking processing. In this way, the accuracy of tracking processing using line of sight information can be improved by making it easier for the user's line of sight to point to the subject while reducing the complexity of the information displayed on the display device 10. The user can be made aware that the line of sight information is being used for tracking processing.

[0116] (Third Embodiment)

[0117] In the description of the above embodiment, it is exemplified that the line of sight information is used when tracking a moving object, but in the following description of this embodiment, the line of sight information is used when detecting a face.

[0118] Fig.12 is a block diagram of an electrical configuration in the camera 1 according to the embodiment. Fig.12 In the embodiment, a face detection circuit 208 is provided to replace Figure 2 The tracking processing circuit 207 in.

[0119] Using template data on human faces prestored in the memory unit 4, the face detection circuit 208 detects a portion (region) of the frame image whose similarity to the template exceeds a reference value as a face image. The face detection method is not limited to the above, and various known techniques can be applied.

[0120] Fig.13 1 is a flowchart for explaining the face detection processing according to the embodiment. When the CPU 3 executes the program stored in the memory unit 4, the processing steps in the flowchart are performed. According to this embodiment, the processing in the flowchart starts when the power of the camera is turned on. Figure 8 The same processing steps as those shown are denoted by the same reference numerals, and a description thereof will not be provided.

[0121] In step S1304, the CPU 3 determines whether the face GUI and the sight line GUI indicating the face position overlap with each other. If they overlap, the process proceeds to step S806, and if they do not overlap, the process proceeds to step S805. For example, when the line as the face GUI and the line as the sight line GUI cross or one of the areas of the face GUI and the area of ​​the sight line GUI includes at least a part of the other area, the face GUI and the sight line GUI overlap. Since the face GUI is displayed in step S1313 to be described, if the face GUI is not displayed (for example, immediately after the process starts), the process proceeds to step S805.

[0122] In step S1307, the CPU 3 sets the threshold Th to the threshold Th1. In step S1308, the CPU 3 sets the threshold Th to the threshold Th2 (<threshold Th1). According to the embodiment, the face GUI is erased (hidden) by the described processing when the face reliability is equal to or less than the threshold Th, so as to prevent the line of sight from being turned to the face GUI that is deviated from the face. Therefore, the threshold Th (threshold Th1) in the case of detecting the viewpoint is greater than the threshold Th (threshold Th2) in the case of not detecting the viewpoint. In this way, when the viewpoint is detected, the start of the deviation of the face GUI can be detected, and the line of sight can be easily directed to the face. In the following description of the embodiment, the threshold Th1 is 70% and the threshold Th2 is 50%.

[0123] In step S1309, the CPU 3 controls the face detection circuit 208 to detect a face. Various existing face detection methods can be applied. For example, in the image of the current frame, a matching process is performed using a reference image, and an area having the highest similarity (correlation) with the reference image is extracted. The reference image is, for example, a template image of a face pre-stored in the memory unit 4. The CPU 3 also controls the face detection circuit 208 to calculate face reliability based on the similarity corresponding to the detected area. The face reliability increases as the similarity increases.

[0124] In step S1310, the CPU 3 sets the result in step S1309 (detection result of the face area) as face position information. The face position information is used as a distance measurement point, for example.

[0125] In step S1311, the CPU 3 determines whether the face reliability is lower than the threshold value Th. If the face reliability is lower than the threshold value Th, the process proceeds to step S1312, and if it is not lower than the threshold value Th, the process proceeds to step S1313. Here, the case where the face reliability is lower than the threshold value Th is the case where the face position and the position of the face GUI are offset from each other. The case where the face reliability is higher than the threshold value Th is the case where the face position and the position of the face GUI are roughly coincident.

[0126] In step S1312, the CPU 3 hides the face GUI. This is because there is a high probability that the face position and the position of the face GUI are far from each other. It is only necessary to prevent the line of sight from going to the face GUI, so the face GUI can be made less noticeable by making the face GUI lighter in color or smaller in size. Similar to the first embodiment, it is preferable to change the display appearance of the OSD (to make it less noticeable). When the face GUI is hidden and only the line of sight GUI is displayed, it is also possible to make the user more aware that face detection processing is being performed through line of sight input.

[0127] In step S1313, the CPU 3 displays the face GUI. This is because there is a high possibility that the position of the face and the position of the face GUI roughly coincide with each other.

[0128] In step S1314, the CPU 3 updates the facial position information based on the line of sight information. For example, the facial recognition area is corrected by weighted addition of the position of the line of sight and the position of the area recognized as the face (face recognition area) and reducing the offset between the face recognition area and the face. Note that the area including the line of sight position (the area corresponding to the line of sight GUI) can be used as the facial recognition area as it is.

[0129] As described above, according to this embodiment, when the sight line detection function is used, the threshold of face reliability is increased, so that the start of the offset between the face GUI and the face position can be quickly detected, and the face GUI can be hidden. In this way, because the sight line is more easily turned to the face, the accuracy of the face detection processing using the sight line information can be improved.

[0130] (Variation Example)

[0131] In the description of the above embodiment, in order to make the GUI less conspicuous, the GUI is hidden, reduced in size or lightened in color, but these methods are not limited to the above. For example, the GUI can be made less conspicuous by changing the brightness or color according to the background (so that the brightness or color is closer to the brightness or color of the background) or by increasing the transmittance (transparency) of the GUI.

[0132] In the description of the above embodiment, it is illustrated that when the subject GUI and the sight line GUI indicating the sight line position overlap, the subject GUI can be made conspicuous and the sight line GUI can be made less conspicuous. However, the subject GUI can be made less conspicuous to make the sight line GUI more conspicuous. Specifically, when the subject GUI and the sight line GUI indicating the sight line position overlap, the subject GUI can be made less conspicuous, for example, by changing the brightness, color, or size.

[0133] In the description of the above embodiment, when the camera performs subject tracking processing or face detection processing, the subject GUI and face GUI are made less obvious so that it is easier to direct the user's line of sight to the desired area (the area including the subject or the face). However, the present invention can also be applied to a case where tracking or face detection processing is not performed. For example, if a GUI image is displayed in an area different from the desired area when the user's viewpoint (line of sight position) in the display device is detected, the user's eyes may turn to the GUI image. Therefore, in order to make it easier for the user's line of sight to be directed to the desired area, when the user's viewpoint (line of sight position) is detected, the GUI image can be made less obvious compared to when the viewpoint is not detected. For example, it can be considered that the present invention relates to an electronic device including the following display control unit and detection unit. Here, the display control unit displays an input image and a GUI image on the display unit. The detection unit detects the user's viewpoint in the input image. When the detection unit detects the viewpoint, the display control unit makes the GUI image less obvious compared to when the detection unit does not detect the viewpoint.

[0134] In the description of the above embodiment, the present invention is illustrated as being applied to a camera, but there are other applications. For example, the present invention can be applied to any electronic device that can receive a line of sight input. For example, the present invention can be applied to a personal computer, a PDA, a mobile phone terminal, a display, or an HMD (head mounted display).

[0135] (Other embodiments)

[0136] The present invention can also be implemented by supplying a program that implements one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer in the system or device to read and execute the program. The present invention can also be implemented by a circuit (e.g., a dedicated ASIC) that implements one or more functions.

[0137] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present invention. Therefore, the appended claims publicly disclose the scope of the present invention.

[0138] This application claims the benefit of Japanese Patent Application No. 2019-162729, filed on September 6, 2019, which is hereby incorporated by reference herein in its entirety.

[0139] [reference numerals list]

[0140] 1 Camera

[0141] 3 CPU

Claims

1. An electronic device, comprising: a display control unit configured to display an input image, a first GUI image, and a second GUI image on a display; a first detection unit configured to detect a position of a predetermined object in the input image; a second detection unit configured to detect a user's viewing point in the input image; and a determination unit configured to determine whether the first GUI image and the second GUI image overlap; wherein, the display control unit displays the first GUI image at a position corresponding to the position of the predetermined object detected by the first detection unit, and displays the second GUI image at a position corresponding to the viewing point detected by the second detection unit, compared with a case where it is determined that the first GUI image and the second GUI image do not overlap, in a case where it is determined that the first GUI image and the second GUI image overlap, the display control unit makes the second GUI image less prominent, and in a case where the position of the predetermined object is detected based on the viewing point, the display control unit does not display the second GUI image.

2. The electronic device according to claim 1, wherein, the first detection unit uses frame images forming the input image to track the predetermined object.

3. The electronic device according to claim 1, wherein, the predetermined object is a face.

4. The electronic device according to claim 1, wherein, in a case where the reliability of the position of the predetermined object detected not based on the viewing point is lower than a threshold, the first detection unit detects the position of the predetermined object based on the viewing point.

5. The electronic device according to claim 4, wherein, the first detection unit does not extract the position of the object based on the viewing point, in a case where the reliability is lower than the threshold, the first detection unit corrects the extracted position of the object based on the viewing point, and detects the corrected position of the object as the position detected based on the viewing point, and in a case where the reliability is higher than the threshold, the first detection unit detects the extracted position of the object as the position not detected based on the viewing point without correcting the extracted position of the object based on the viewing point.

6. The electronic device according to claim 4, wherein, a threshold in a case where the second detection unit detects the viewing point is greater than a threshold in a case where the viewing point is not detected.

7. The electronic device according to claim 1, wherein, in a case where it is determined that the first GUI image and the second GUI image overlap, the display control unit changes at least one of brightness, color, size, and transparency of the first GUI image such that the first GUI image is more prominent than the second GUI image.

8. The electronic device according to claim 1, wherein, In a case where it is determined that the first GUI image and the second GUI image overlap, the display control unit changes at least one of the brightness, color, size, and transparency of the second GUI image so that the second GUI image is less noticeable than the first GUI image.

9. The electronic device according to claim 1, wherein, in a case where it is determined that the first GUI image and the second GUI image overlap, the display control unit does not display the second GUI image.

10. The electronic device according to claim 4, wherein, in a case where the reliability is lower than the threshold, the display control unit displays the second GUI image indicating the user's viewing point in the input image, and in a case where the reliability is higher than the threshold, does not display the second GUI image indicating the user's viewing point in the input image.

11. An electronic device, comprising: a setting unit configured to set a position corresponding to the position of a predetermined object as a position for displaying a first GUI image; a display control unit configured to display the first GUI image and a second GUI image at a display, the second GUI image being based on detection of a user's viewing point; and a determination unit configured to determine whether the first GUI image and the second GUI image overlap, where compared with a case where it is determined that the first GUI image and the second GUI image do not overlap, in a case where it is determined that the first GUI image and the second GUI image overlap, the display control unit makes the second GUI image less noticeable, and in a case where the position of the predetermined object is detected based on the viewing point, the display control unit does not display the second GUI image.

12. A method for controlling an electronic device, comprising: a display control step for displaying an input image, a first GUI image, and a second GUI image at a display; a first detection step for detecting the position of a predetermined object in the input image; a second detection step for detecting a user's viewing point in the input image; and a determination step for determining whether the first GUI image and the second GUI image overlap; wherein, in the display control step, the first GUI image is displayed at a position corresponding to the position of the predetermined object detected in the first detection step, and the second GUI image is displayed at a position corresponding to the viewing point detected in the second detection step, in the display control step, compared with a case where it is determined that the first GUI image and the second GUI image do not overlap, in a case where it is determined that the first GUI image and the second GUI image overlap, the second GUI image is made less noticeable, and in the display control step, in a case where the position of the predetermined object is detected based on the viewing point, the second GUI image is not displayed.

13. A method for controlling an electronic device, comprising: A setting step for setting a position corresponding to the position of a predetermined object as a position for displaying a first GUI image; A display control step for displaying the first GUI image and a second GUI image at a display, the second GUI image being based on detection of a user's viewpoint; And A determination step for determining whether the first GUI image and the second GUI image overlap, wherein In the display control step, compared with a case where it is determined that the first GUI image and the second GUI image do not overlap, in a case where it is determined that the first GUI image and the second GUI image overlap, the second GUI image is made less visible, and In the display control step, in a case where the position of the predetermined object is detected based on the viewpoint, the second GUI image is not displayed.

14. A computer-readable storage medium storing a program for causing a computer to function as each unit of the electronic device according to any one of claims 1 to 11.

15. A computer program product including a program for causing a computer to function as each unit of the electronic device according to any one of claims 1 to 11.

Images