Electronic device, electronic device control method, and recording medium

Abstract

To continue to capture a subject intended by a user from before the start of a zoom operation even when stopping selection of a subject based on line-of-sight detection after the start of the zoom operation.SOLUTION: An electronic apparatus includes: line-of-sight detection means that detects a user's line of sight; zoom operation means that performs zoom operation; subject detection means that detects a subject from an image picked up by imaging means; selection means that selects the subject based on the position of the line of sight; and control means that, when a specific function is executed for the selected subject, in response to the start of the zoom operation by the zoom operation means, stops selection of the subject based on a line-of-sight position, and controls to continuously execute the specific function for a subject selected based on the line-of-sight position before the start of the zoom operation.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an electronic device and a control method for an electronic device. allusion to law and recording media. [Background technology]

[0002] In recent years, electronic devices that allow users to operate them with their own gaze (hereinafter simply referred to as "gaze input") have become widespread. Electronic devices (for example, imaging devices such as digital cameras, game consoles) equipped with an gaze detection device that realizes an gaze input function are effective as a means for quickly issuing operation instructions, particularly when a user wants to issue operation instructions to the electronic device immediately. As a related technology, the technology disclosed in Patent Document 1 has been proposed. In the technology disclosed in Patent Document 1, when a camera equipped with an gaze detection device determines that the photographing lens is in the process of zooming, gaze detection is stopped and gaze detection information accumulated before the zooming operation is reset. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 3-293367 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology of Patent Document 1, because gaze detection is stopped during zooming, detection of the position of the main subject selected by gaze input before the zooming operation is also stopped, resulting in a problem in that the user may miss an opportunity to capture a photo of the main subject they desired during or immediately after zooming.

[0005] Furthermore, in an imaging device equipped with a gaze detection device, the position of the subject in the captured image changes quickly during zooming, making it difficult for the user to keep their gaze on the intended subject, resulting in gaze blur. This causes the gaze position to shift from the intended subject, resulting in the problem that the user cannot continue to capture the intended subject.

[0006] To provide a method for continuously capturing a subject that a user intended to capture before starting a zoom operation, even when selection of the subject by line-of-sight detection is stopped after starting a zoom operation. [Means for solving the problem]

[0007] In order to solve the above problems, the electronic device of the present invention includes a line-of-sight detection unit that detects the line of sight of a user, and a zoom operation unit that performs a zoom operation. And, before a selection means for selecting a subject from the image based on the position of the line of sight; before In response to the zoom operation being started by the zoom operation means, The detection of the line of sight by the line of sight detection means is not stopped, and the selection of the subject based on the position of the line of sight is stopped. and a control means for controlling the temperature to be higher than the predetermined temperature. [Effects of the Invention]

[0008] According to the present invention, even if selection of a subject by gaze detection is stopped after a zoom operation is started, it is possible to continue capturing the subject that the user intended before the zoom operation was started. [Brief explanation of the drawings]

[0009] [Figure 1] 1A and 1B are diagrams showing an example of the external configuration of an imaging device according to a first embodiment of the present invention, in which FIG. 1A is a front perspective view of the imaging device as seen from the front side, and FIG. 1B is a rear perspective view of the imaging device as seen from the rear side. [Figure 2] 2 is a block diagram showing an example of the overall configuration of the imaging device in FIG. 1 when an interchangeable lens type lens unit is attached to the imaging device. FIG. [Figure 3]4 is a flowchart showing the flow of control processing performed by the imaging device according to the first embodiment. [Figure 4] 10 is a flowchart showing the flow of gaze return trigger processing performed by the imaging device according to the first embodiment. [Figure 5] 10 is a flowchart showing the flow of gaze return trigger processing performed by the imaging device according to the second embodiment. [Figure 6] 11 is a flowchart showing the flow of gaze return trigger processing performed by the imaging device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0011] First Embodiment A first embodiment of the present invention will be described below with reference to the drawings. The electronic device according to the first embodiment of the present invention is an imaging device 100 as an example of an electronic device to which the present invention can be applied. Fig. 1 shows an example of the external configuration of the imaging device 100 according to the first embodiment of the present invention, with Fig. 1(A) being a front perspective view of the imaging device 100 as seen from the front side, and Fig. 1(B) being a rear perspective view of the imaging device as seen from the rear side.

[0012] 1(A) and 1(B), the imaging device 100 according to the first embodiment of the present invention is an interchangeable lens imaging device 100, such as a digital mirrorless single-lens camera, a digital single-lens reflex camera, or other interchangeable lens digital camera. The imaging device 100 according to the first embodiment of the present invention is equipped with an eye-gaze input function.

[0013] A display unit 28 is provided on the rear surface of the imaging device 1, and is a display device for displaying image data and various information. A touch panel 70a can detect touch operations on the display surface (touch operation surface) of the display unit 28. An extra-finder display unit 43 is a display provided on the top surface of the imaging device 100, and displays various settings of the imaging device 100, such as shutter speed and aperture. The shutter button 61 is an operation member for issuing shooting instructions. The mode selector switch 60 is an operation member for switching between various modes. The terminal cover 40 is a cover for protecting a connector (not shown) that connects the imaging device 100 to a connection cable for connecting an external device. The main electronic dial 71 is a rotary operation member; the user can change settings such as shutter speed and aperture by rotating the main electronic dial 71. The power switch 72 is an operation member for switching the power of the imaging device 100 on and off. The sub electronic dial 73 is a rotary operation member, and the user can perform operations such as moving the selection frame (cursor) and scrolling through images by rotating the sub electronic dial 73. The four-way key 74 is configured so that each of the up, down, left, and right sections can be pressed, and processing can be performed according to the section of the four-way key 74 that is pressed. The SET button 75 is a push button, and is mainly used to confirm a selection item.

[0014] The video button 76 is used to start or stop video shooting (video recording). The AE lock button 77 is a push button, and pressing the AE lock button 77 in a shooting standby state fixes the exposure state. The enlarge button 78 is an operation button for switching the enlargement mode on and off in the live view display (hereinafter also simply referred to as "LV display") in the shooting mode. After turning on the enlargement mode with the enlargement button 78, the live view image (LV image) can be enlarged or reduced by operating the main electronic dial 71. The enlargement button 78 also functions as an operation button for enlarging the image displayed on the display unit 28 (i.e., the playback image) and increasing the magnification ratio of the playback image in the playback mode.

[0015] The playback button 79 is an operation button for switching between shooting mode and playback mode. Pressing the playback button 79 during shooting mode switches to playback mode, and the latest image data among the image data recorded (saved) on the recording medium 200 (described later) can be displayed on the display unit 28. The menu button 81 is a push button used to perform an instruction operation to display a menu screen, and when the menu button 81 is pressed, a menu screen on which various settings can be made is displayed on the display unit 28. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four-way key 74, and the SET button 75.

[0016] The communication terminal 10 is a communication terminal through which the imaging device 100 communicates with a detachable lens unit 150 (described later). The eyepiece 16 is the eyepiece of the eyepiece finder 17 (a peer-type finder), and the user can view an image displayed on an internal electronic viewfinder 29 (hereinafter simply referred to as "EVF 29") through the eyepiece 16. The EVF 29 as a display device is not shown in FIG. 1 but is shown in FIG. 2. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether the user (photographer) has placed their eye on the eyepiece 16. The lid 202 is a lid for a slot for storing the recording medium 200. The grip unit 90 is a holding unit shaped to be easily gripped with the user's right hand when holding the imaging device 100. The shutter button 61 and main electronic dial 71 are located at positions that can be operated with the index finger of the right hand when the user holds the imaging device 100 by gripping the grip unit 90 with the little finger, ring finger, and middle finger of the right hand. The sub electronic dial 73 and touch bar 82 are located at positions that can be operated with the thumb of the right hand when the user holds the imaging device 100 by gripping the grip unit 90 with the little finger, ring finger, and middle finger of the right hand. The thumb rest 91 (thumb standby position) is a grip member provided on the back side of the imaging device 100 in a position where it is easy to place the thumb of the right hand that is gripping the grip unit 90 when none of the operation members are being operated. The thumb rest 91 is made of a rubber member or the like to increase holding strength (grip feeling).

[0017] Next, the overall configuration of the imaging device when an interchangeable lens unit 150 is attached to the imaging device 100 according to the first embodiment of the present invention shown in Fig. 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the overall configuration of the imaging device when the lens unit 150 is attached to the imaging device 100 shown in Fig. 1.

[0018] As shown in FIG. 2, the lens unit 150 is a lens unit equipped with an interchangeable photographic lens. The lens 103 is usually composed of multiple lenses, but FIG. 2 shows only one lens for simplicity. The communication terminal 6 is a communication terminal through which the lens unit 150 communicates with the imaging device 100, and the communication terminal 10 is a communication terminal through which the imaging device 100 communicates with the lens unit 150. The lens unit 150 communicates with the system control unit 50 of the imaging device 100 via the communication terminal 6 and the communication terminal 10. The lens unit 150 controls the aperture 1 via the aperture drive circuit 2 using the internal lens system control circuit 4. The lens unit 150 adjusts the focus by displacing the position of the lens 103 via the AF drive circuit 3 using the lens system control circuit 4. The zoom ring 7a is an operating member that allows the user to perform zoom operations and functions as a zoom operation means. When the user starts a zoom operation, the lens unit 150 detects the presence and amount of rotation of the zoom ring 7a using the zoom ring rotation detection unit 7b, which can detect the rotation of the zoom ring 7a, and performs a zooming operation by controlling the lens 103 via the lens system control circuit 4. Furthermore, when the user finishes the zoom operation, the lens unit 150 detects the absence of rotation of the zoom ring 7a using the zoom ring rotation detection unit 7b, and controls the lens 103 via the lens system control circuit 4 to end the zooming operation.

[0019] The zoom operation means is not limited to the zoom ring 7a, but may be the touch panel 70a. For example, when the user performs a touch operation of pinching in or out on the touch panel 70a, the lens unit 150 may control the lens 103 via the lens system control circuit 4 to perform a zooming operation according to the change in the distance between the touched positions.

[0020] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

[0021] The imaging unit 22 is an imaging element (image sensor) configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 22 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 23 converts the analog signal output from the imaging unit 22 into a digital signal.

[0022] The image processing unit 24 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data from the A / D converter 23 or data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This results in TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, EF (flash pre-flash) processing, etc. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (auto white balance) processing based on the arithmetic results obtained.

[0023] The output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15. Alternatively, the output data from the A / D converter 23 is written to the memory 32 via the memory control unit 15 without going through the image processing unit 24. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, as well as image data to be displayed on the display unit 28 and the EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still image data frames, moving image data for a predetermined period of time, and audio data.

[0024] The memory 32 also serves as a memory (video memory) for image display. The D / A converter 19 converts image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28 or the EVF 29. The display image data written to the memory 32 in this way is displayed on the display unit 28 or the EVF 29 via the D / A converter 19. The display unit 28 and the EVF 29 are each a display such as an LCD or an organic EL, and perform display according to the analog signal from the D / A converter 19. The digital signal that has been A / D converted by the A / D converter 23 and stored in the memory 32 is converted into an analog signal by the D / A converter 19, and the analog signal is sequentially transferred to and displayed on the display unit 28 or the EVF 29, thereby performing live view display (LV display). Hereinafter, an image displayed in live view display on the display unit 28 or the EVF 29 will be referred to as a live view image (LV image).

[0025] The system control unit 50 is a control unit including at least one processor and / or at least one circuit, and controls the entire imaging device 100. The system control unit 50 is both a processor and a circuit. The system control unit 50 executes programs stored in a nonvolatile memory 56 to realize the processes of the embodiments of the present invention, which will be described later. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 19, the display unit 28, the EVF 29, etc.

[0026] The system memory 52 is, for example, a RAM, and the system control unit 50 loads constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like into the system memory 52.

[0027] The nonvolatile memory 56 is an electrically erasable and recordable memory, such as an EEPROM. Constants, programs, etc. for the operation of the system control unit 50 are stored in the nonvolatile memory 56. The programs referred to here are programs for executing various flowcharts described later in each embodiment of the present invention.

[0028] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0029] The communication unit 54 transmits and receives video signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit image data (including LV image data) captured by the imaging unit 22 as an imaging means and image data recorded on the recording medium 200, and can receive image data and various other information from external devices.

[0030] The orientation detection unit 55 detects the orientation of the image capture device 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is possible to determine whether an image captured by the image capture unit 22 was captured with the image capture device 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the image capture unit 22, or rotate and record the captured image. An acceleration sensor, a gyro sensor, or the like can be used as the orientation detection unit 55. The acceleration sensor or gyro sensor of the orientation detection unit 55 can also be used to detect movement of the image capture device 100 (panning, tilting, lifting, whether the image capture device 100 is stationary, etc.).

[0031] The eyepiece detection unit 57 is an eyepiece detection sensor that detects (proximity detection) whether an eye (object) approaches (approach) or moves away (away) from the eyepiece 16 of the eyepiece viewfinder 17 (hereinafter also simply referred to as the "viewfinder"). The system control unit 50 switches the display unit 28 and the EVF 29 between on (display state) and off (non-display state) depending on the state detected by the eyepiece detection unit 57. More specifically, at least in a shooting standby state and when the display destination switching setting is automatic switching, when the eye is not in contact with the camera, the display is turned on with the display as the display unit 28 and the EVF 29 is hidden. When the eye is in contact with the camera, the display is turned on with the display as the EVF 29 and the display unit 28 is hidden. For example, an infrared proximity sensor can be used as the eyepiece detection unit 57, and it can detect the approach of an object to the eyepiece 16 of the viewfinder 17 that incorporates the EVF 29. When an object approaches, infrared light emitted from a light-emitting unit (not shown) of the eyepiece detection unit 57 is reflected by the object and received by a light-receiving unit (not shown) of the infrared proximity sensor. The amount of received infrared light can also determine the distance the object is approaching the eyepiece 16 (eyepiece distance). In this way, the eyepiece detection unit 57 performs eyepiece detection, which detects the proximity of an object to the eyepiece 16. When an object approaching within a predetermined distance from the eyepiece 16 is detected from a non-eyepiece state (non-approach state), it is detected as being in eye contact. When an object detected as approaching moves away from the eyepiece state (approach state) by more than a predetermined distance, it is detected as being away from the eye. The threshold for detecting eye contact and the threshold for detecting eye separation may be different, for example, by providing hysteresis. Furthermore, after eye contact is detected, the eye is considered to be in the eye contact state until eye separation is detected. After eye separation is detected, the eye is considered to be in the non-eye contact state until eye contact is detected. The infrared proximity sensor is one example of an eye-contact detection sensor (eye-contact detection unit 57), and other sensors may be used as the eye-contact detection sensor (eye-contact detection unit 57) as long as they can detect a state that can be considered as eye-contact.

[0032] Various setting values of the image capturing apparatus 100, such as shutter speed and aperture, are displayed on the outside viewfinder display section 43 via an outside viewfinder display section drive circuit 44.

[0033] The power supply control unit 80 is composed of a battery detection circuit (not shown), a DC-DC converter (not shown), a switch circuit (not shown) for switching between blocks to which electricity is applied, etc., and detects whether a battery is installed, the type of battery, and the remaining battery charge. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or Li batteries, an AC adapter, etc.

[0034] The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images (image data), and is configured from a semiconductor memory, a magnetic disk, or the like.

[0035] The operation unit 70 is an input unit that accepts operations by the user (hereinafter also simply referred to as "user operations") and is used to input various operational instructions to the system control unit 50. As shown in Fig. 2, the operation unit 70 has a shutter button 61, a mode selector switch 60, a power switch 72, a touch panel 70a, and other operation members 70b. The other operation members 70b also have a main electronic dial 71, a sub electronic dial 73, a four-way key 74, a SET button 75, a video button 76, an AE lock button 77, a magnification button 78, a playback button 79, a menu button 81, a touch bar 82, and the like.

[0036] The shutter button 61 has a first shutter switch 62 and a second shutter switch 64. When the shutter button 61 is pressed halfway (a shooting preparation command), the first shutter switch 62 is turned on and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0037] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The second shutter switch signal SW2 causes the system control unit 50 to start a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file.

[0038] The mode selector switch 60 switches the operating mode of the system control unit 50 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for specific capture scenes. The user can directly switch to one of these modes using the mode selector switch 60. Alternatively, the user may first use the mode selector switch 60 to switch to a list screen of capture modes, and then use other operating members to selectively switch to one of the multiple modes displayed on the list screen of capture modes. Similarly, the video capture mode may also include multiple modes.

[0039] The touch panel 70a is a touch sensor that detects various touch operations on the display surface of the display unit 28 (the operation surface of the touch panel 70a). The touch panel 70a and the display unit 28 can be integrally configured. For example, the touch panel 70a is configured so that its light transmittance does not interfere with the display of the display unit 28, and is attached to the upper layer of the display surface of the display unit 28. Input coordinates on the touch panel 70a correspond to display coordinates on the display surface of the display unit 28. This makes it possible to provide a GUI (Graphical User Interface) that allows the user to directly operate the screen displayed on the display unit 28.

[0040] The gaze detection unit 160 is an eye tracking unit that detects the gaze of a user who has placed their eyepiece 16 close to the EVF 29, and if so, where they are looking. That is, the gaze detection unit 160 detects the direction and position of the user's gaze and functions as a receiving unit (gaze input unit) that receives gaze input based on the detection results. The gaze detection unit 160 also includes a dichroic mirror 162, an imaging lens 163, a gaze detection sensor 164, an infrared light emitting diode 166, and a gaze detection circuit 165.

[0041] The infrared light emitting diode 166 is a light emitting element that emits infrared light onto the user's eyeball (eye) 161 placed in the eyepiece 16. The infrared light emitted from the infrared light emitting diode 166 is reflected by the eyeball (eye) 161, and the reflected infrared light reaches the dichroic mirror 162. The dichroic mirror 162 reflects only the infrared light and transmits visible light. The reflected infrared light, whose optical path has been changed, forms an image on the imaging surface of the line of sight detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical component that constitutes the line of sight detection optical system. The line of sight detection sensor 164 is made up of an imaging element such as a CCD image sensor. The line of sight detection sensor 164 photoelectrically converts the incident reflected infrared light into an electrical signal and outputs it to the line of sight detection circuit 165. The gaze detection circuit 165 has at least one processor, and detects the gaze position of the user from the image or movement of the user's eyeball (eye) 161 based on the output signal of the gaze detection sensor 164, and outputs the detection information to the system control unit 50.

[0042] In the first embodiment, the gaze detection unit 160 detects the user's gaze by using a gaze detection method known as the corneal reflex method. The corneal reflex method is a gaze detection method that detects the direction and position of the gaze from the positional relationship between the light reflected by the eyeball (eye) 161, particularly the cornea, of infrared light emitted from the infrared light-emitting diode 166 and the pupil of the eyeball (eye) 161. However, the gaze detection method that can be used in the present invention is not limited to the corneal reflex method described above, and any gaze detection method other than the corneal reflex method may be used as long as it can detect the direction and position of the gaze. For example, a gaze detection method known as the scleral reflex method, which utilizes the difference in light reflectance between the iris and white of the eye, may be used.

[0043] As described above, in the first embodiment, the light-projecting unit and light-receiving unit of eye proximity detection unit 57 are separate devices from infrared light-emitting diode 166 and line-of-sight detection sensor 164. However, the present invention is not limited to this, and infrared light-emitting diode 166 may also serve as the light-projecting unit of eye proximity detection unit 57, and line-of-sight detection sensor 164 may also serve as the light-receiving unit of eye proximity detection unit 57.

[0044] The system control unit 50 can detect the following operations or states based on the output from the line-of-sight detection unit 160 (that is, the detection information received from the line-of-sight detection circuit 165). The gaze of the user who has placed his / her eye on the eyepiece unit 16 is newly input (detected). In other words, gaze input begins. The eyepiece 16 is receiving gaze input from the user. The user is gazing at the eyepiece 16. The user who has placed his / her eye on the eyepiece 16 moves his / her gaze away from the eyepiece 16. In other words, the gaze input ends. The user is not making any eye-gaze input to the eyepiece 16.

[0045] The term "gazing" as used herein means that the user continues to look at approximately the same position for a certain period of time. The system control unit 50 determines whether the user is gazing by, for example, determining that the user is gazing when the user's gaze position does not exceed a predetermined amount of movement for a first predetermined period of time (e.g., approximately 0.5 seconds). The first predetermined period of time may be a time that can be set by the user, a predetermined fixed period of time, or may vary depending on the distance between the previous and current gaze positions. For example, the system control unit 50 determines that the user is gazing when, based on the detection information received from the gaze detection circuit 165, the duration of a state in which the user's gaze is detected at approximately the same position (hereinafter referred to as a "no gaze movement state") exceeds a predetermined threshold period. Furthermore, the system control unit 50 determines that the user is gazing when, for example, the average position of the gaze detection positions over a short period of time (≦predetermined threshold period) including the most recent gaze detection timing falls within a predetermined range and the variation (variance) is less than a predetermined value.

[0046] As described above, in the imaging device 100 according to the first embodiment of the present invention, the combination of the EVF 29 and the gaze detection unit 160 realizes the gaze input function.

[0047] Next, a control process performed by the imaging device according to the first embodiment will be described. In the first embodiment, the system control unit 50 in the imaging device 100, as a control means, performs a control process of switching from subject selection based on the user's line of sight to subject detection using a predetermined algorithm of the imaging device 100 in response to the start of a zoom operation. Each step in the control process, including a gaze return trigger process (to be described later), is realized by the system control unit 50 loading a program stored in the nonvolatile memory 56 into the system memory 52 and executing it when the imaging device 100 is powered on by the power switch 72.

[0048] FIG. 3 is a flowchart showing the flow of control processing performed by the imaging device 100 according to the first embodiment. Hereinafter, subject selection based on the user's line of sight will be referred to as "gaze-based subject selection" or simply as "subject selection." Furthermore, subject detection using a predetermined algorithm in the imaging device 100 will be simply referred to as "subject detection." The "predetermined algorithm" referred to in the present invention refers to an algorithm capable of detecting a subject. For example, the predetermined algorithm may be an algorithm utilizing deep learning. The imaging device 100 is equipped with a gaze-based subject selection function and a subject detection function. The gaze-based subject selection function is a function for selecting a subject located at the gaze position indicated by a gaze pointer (described later). Furthermore, the subject detection function is a function for detecting a subject using a predetermined algorithm, and the detected subject is displayed in an AF frame. In the following description, "S" refers to a step.

[0049] First, in S301, the system control unit 50, which serves as a subject detection unit, starts a subject detection process for detecting a subject using a predetermined algorithm of the imaging device 100 by turning on the subject detection function.

[0050] Next, in S302, the system control unit 50 turns on the gaze-based subject selection function and starts gaze-based subject selection. Also, by turning on the display of the gaze pointer on the EVF 29, the system control unit 50 causes the EVF 29 to display the gaze pointer in a first predetermined color. The gaze pointer here is a mark indicating the user's gaze position detected by the gaze detection unit 160 (hereinafter also simply referred to as the "gaze position"). In other words, the gaze pointer indicates the gaze position based on the gaze input received by the gaze detection unit 160 as a gaze input means. In other words, when the gaze-based subject selection function is turned on in S302, the gaze pointer indicating the gaze position detected by the gaze detection unit 160 is displayed in a first predetermined color on the EVF 29. For example, the first predetermined color may be red.

[0051] Next, in S303, the system control unit 50 determines whether or not a subject detected by subject detection is present at the gaze position indicated by the gaze pointer displayed in a first predetermined color (i.e., the user's intended gaze position). If the system control unit 50 determines in S303 that a subject detected by subject detection is present at the user's intended gaze position, the system control unit 50 proceeds to S305. On the other hand, if the system control unit 50 determines in S303 that a subject detected by subject detection is not present at the user's intended gaze position, the system control unit 50 proceeds to S304.

[0052] In S304, the system control unit 50 sets the object detection flag to OFF and then proceeds to S306a. The object detection flag indicates whether or not an object detected by object detection is present at the user's intended gaze position. In S306a, the system control unit 50 executes a specific function at the user's intended gaze position and proceeds to S307. The specific function here refers to functions such as AF, AE, AWB, and object tracking. Since the determination in S303 is No, no object is detected at the user's intended gaze position. Therefore, the specific function is executed at a position determined to be the detected gaze of the user (the user's intended gaze position). Since no object is present at the user's intended gaze position, the specific function is executed on a portion of the live view image where no object is recognized, such as the background or an object. In this case, the area where the specific function is executed may be a predetermined area, or may be an area determined to be the same distance from the position specified by the gaze position to the imaging unit 22. Determining the area based on distance is convenient when, for example, a subject is present but not determined to be a subject, or when photographing scenery.

[0053] On the other hand, in S305, the system control unit 50 sets the subject detection flag to ON, and then proceeds to S306b. In S306b, the system control unit 50 executes a specific function on the subject at the detected gaze position (the subject at the gaze position intended by the user), that is, executes the specific function on the subject at the gaze position, and then proceeds to S307.

[0054] In S307, the system control unit 50 determines whether the user has started a zoom operation using the operation unit 70 or the zoom ring 7a, and if it determines that a zoom operation has started, the process proceeds to S308. On the other hand, if it determines in S307 that a zoom operation has not started, the system control unit 50 returns the process to S303.

[0055] In S308, the system control unit 50 ends the gaze-based subject selection by turning off the gaze-based subject selection function. The system control unit 50 also turns off the display of the gaze pointer displayed on the EVF 29. That is, the display of the gaze pointer on the EVF 29 in the first predetermined color is also turned off. Here, not only does the gaze-based subject selection end, but the gaze detection process by the gaze detection unit 160 may also be interrupted.

[0056] Next, in S309, the system control unit 50 determines whether the subject detection flag is on, and if it is determined that the subject detection flag is on, proceeds to S310. On the other hand, if it is determined that the subject detection flag is not on in S309, proceeds to S313.

[0057] In S310, the system control unit 50 determines whether or not the live-view image contains a subject that the user selected with their line of sight before the zoom operation (hereinafter simply referred to as a "subject selected with their line of sight"). In S310, if the system control unit 50 determines that the live-view image contains a subject selected with their line of sight, the process proceeds to S312. On the other hand, if the system control unit 50 determines that the live-view image does not contain a subject selected with their line of sight, the process proceeds to S311.

[0058] In S311, the system control unit 50 sets the subject detection flag to OFF, and then the process proceeds to S313.

[0059] In S312, the system control unit 50 selects a subject by subject detection, i.e., selects the subject detected by subject detection (the subject selected by the line of sight in the live view image), and proceeds to S314. Subject detection is performed, for example, by pattern matching processing with the image of the subject selected by the line of sight. As a result, the subject selected by the line of sight continues to be detected from the image even while the subject selection by the line of sight is interrupted, and becomes the target for executing a specific function.

[0060] In S313, the system controller 50 stops the execution of the specific function that started in S306a or S306b, and proceeds to S314.

[0061] In S314, the system control unit 50 determines whether the user's zoom operation has ended, and if it determines that the user's zoom operation has ended, the process proceeds to S315. On the other hand, if the system control unit 50 determines that the user's zoom operation has not ended in S314, the process returns to S309.

[0062] In S315, the system control unit 50 determines whether the object detection flag is on, and if it is determined that the object detection flag is on, the process proceeds to S316. On the other hand, if it is determined that the object detection flag is not on in S315, the system control unit 50 proceeds to S317.

[0063] In S316, the system control unit 50 performs gaze return trigger processing, and if it determines that the gaze return trigger condition is satisfied, ends the gaze return trigger processing and proceeds to S317 to return to the gaze-based subject selection function. The gaze return trigger processing will be described in detail later.

[0064] In S317, the system control unit 50 starts selecting a subject by gaze by turning on the subject selection function by gaze, and also turns on the display of the gaze pointer on the EVF 29 to display the gaze pointer on the EVF 29 in a first predetermined color.

[0065] Next, in S318, the system control unit 50 determines whether the shooting mode has ended, and if it determines that the shooting mode has not ended, the process returns to S303. On the other hand, if it determines that the shooting mode has ended in S318, the system control unit 50 ends the process.

[0066] As described above, the control process performed by the imaging device 100 according to the first embodiment can perform the following controls. In the present invention, "before a zoom operation is started" is referred to as "before a zoom operation," "while a zoom operation is being performed" is referred to as "during a zoom operation," and "after a zoom operation has ended" is referred to as "after a zoom operation." If a subject is detected at the gaze position by subject detection before zooming, and if a subject is also detected by subject detection during zooming, the gaze return trigger condition will be met after zooming, and the gaze-based subject selection function will return. If no subject is detected at the gaze position by subject detection before zooming, and a specific function is executed at the gaze position, the execution of the specific function that started execution will be stopped during zooming. Once zooming is completed, the function will return to gaze-based subject selection function. The gaze-based subject selection function will be turned off during zoom operations, regardless of whether or not a subject was detected in the gaze position by subject detection before zoom operations. If a subject is detected at the gaze position by subject detection before zooming, but the subject can no longer be detected by subject detection during zooming, the execution of a specific function that has started execution will stop. Once zooming is completed, the function will return to gaze-based subject selection. The specific functions are not limited to the above-mentioned AF, AE, AWB, and subject tracking functions, and may be functions other than the autofocus function, auto exposure function, auto white balance function, and subject tracking function as long as they use subject selection.

[0067] As described above, in an embodiment of the present invention, when a zoom operation is initiated during gaze-based subject selection, the subject selection method switches from gaze-based subject selection to subject detection, and when the gaze return trigger condition is met after the zoom operation, the subject selection method returns to gaze-based subject selection. Therefore, even during a zoom operation, the subject intended by the user before the zoom operation can be continuously captured. In this way, in an embodiment of the present invention, the gaze-based subject selection function is turned off during a zoom operation, which prevents erroneous input (e.g., selection of the wrong subject) due to gaze blurring, and prevents missed photographic opportunities after a zoom operation.

[0068] Next, the gaze return trigger process (the gaze return trigger process performed by the system control unit 50 in S316) performed by the imaging device according to the first embodiment will be described. Fig. 4 is a flowchart showing the flow of the gaze return trigger process in S316 of Fig. 3.

[0069] First, in S401, the system control unit 50 starts the timer and turns on the display of the line-of-sight pointer on the EVF 29, thereby displaying the line-of-sight pointer on the EVF 29 in a second predetermined color. The second predetermined color is a color different from the first predetermined color. For example, if the first predetermined color is red, the second predetermined color may be gray, or a color other than red or gray. Starting the timer in S401 means setting the system timer 53 to a predetermined time and starting measurement by the system timer 53. Note that the setting value (predetermined time) of the system timer 53 may be a time that can be set by the user or may be a predetermined fixed time. For example, the user may select the setting value (predetermined time) of the system timer 53 from a menu screen displayed on the display unit 28 by pressing the menu button 81.

[0070] Next, in S402, the system control unit 50 determines whether the timer started in S401 has expired, and if it determines that the timer has expired, turns off the display of the gaze pointer pointing to the EVF 29 in the second predetermined color and ends the gaze return trigger process. Thereafter, the system control unit proceeds to S317 in FIG. 3.

[0071] On the other hand, if the system control unit 50 determines in S402 that the timer has not expired, it repeats the process of S402 until the timer expires.

[0072] As described above, in the first embodiment, the gaze return trigger condition in the gaze return trigger process is timer expiration. Timer expiration allows the system to return to the gaze-based subject selection function. By using this control, if the timer is set short, the system can select a subject by gaze without waiting a long time after the zoom operation is completed, allowing the system to quickly execute a specific function for the desired subject. If the timer is set long, the system will not execute a specific function due to an unintended movement of the user's gaze position, allowing the user to check the live view image without feeling annoyed. Furthermore, in the first embodiment, in step S401, the system displays the gaze pointer on the EVF 29 in a second predetermined color different from the first predetermined color, thereby intuitively informing the user that the specific function at the gaze position has not yet been executed.

[0073] Second Embodiment A second embodiment of the present invention will now be described.

[0074] The configuration of the imaging device according to the second embodiment of the present invention is similar to the configuration of the imaging device 100 according to the first embodiment shown in Figures 1(A), 1(B) and 2, and therefore will not be described.

[0075] Furthermore, except for the gaze return trigger processing of S316 in Figure 3, the flow of the control processing performed by the imaging device of the second embodiment is similar to the flow of the control processing performed by the imaging device 100 of the first embodiment shown in Figure 3, so explanation will be omitted.

[0076] The following describes the gaze return trigger process performed by the imaging device according to the second embodiment. Fig. 5 is a flowchart showing the gaze return trigger process of S316 in Fig. 3 performed by the imaging device according to the second embodiment.

[0077] First, in S501, the system control unit 50 determines whether the user has operated a specific operation member of the operation unit 70. If it determines that the user has operated a specific operation member, the system control unit 50 ends the gaze return trigger process. After that, the system control unit proceeds to S317 in FIG. 3.

[0078] On the other hand, in S501, if the system control unit 50 determines that the specific operation member has not been operated by the user, it repeats the process of S501 until the specific operation member is operated.

[0079] As described above, in the second embodiment, the gaze return trigger condition in the gaze return trigger process is the operation of a specific operating member (i.e., the user's operation of a specific operating member). The gaze-based subject selection function can be restored in response to the user's clear intention to operate the specific operating member. This prevents the gaze-based subject selection function from being restored at a time unintended by the user, thereby reducing the risk of a subject being selected and a missed photo opportunity. The specific operating member may be a predetermined member or a member determined by the user. For example, the user may select a specific operating member from a menu screen displayed on the display unit 28 by pressing the menu button 81.

[0080] <Third embodiment> A third embodiment of the present invention will now be described.

[0081] The configuration of the imaging device according to the third embodiment of the present invention is similar to the configuration of the imaging device 100 according to the first embodiment shown in Figures 1(A), 1(B) and 2, and therefore will not be described.

[0082] Furthermore, except for the gaze return trigger processing of S316 in Figure 3, the flow of the control processing performed by the imaging device of the third embodiment is similar to the flow of the control processing performed by the imaging device 100 of the first embodiment shown in Figure 3, so explanation will be omitted.

[0083] The following describes the gaze return trigger process performed by the imaging device according to the third embodiment. Fig. 6 is a flowchart showing the gaze return trigger process of S316 in Fig. 3 performed by the imaging device according to the third embodiment.

[0084] First, in S601, the system control unit 50 turns on the display of the gaze pointer on the EVF 29, thereby causing the EVF 29 to display the gaze pointer in a second predetermined color. The second predetermined color is a color different from the first predetermined color. For example, if the first predetermined color is red, the second predetermined color may be gray, or may be a color other than red or gray.

[0085] Next, in S602, the system control unit 50 determines whether the user is gazing at the AF frame for subject tracking in subject detection (i.e., the AF frame determined by the subject detection unit). If the system control unit 50 determines in S602 that the user is gazing at the AF frame, it turns off the display of the gaze pointer in the second predetermined color on the EVF 29 and ends the gaze return trigger process. The system control unit then proceeds to S317 in FIG. 3 . Specifically, in S602, the system control unit 50 determines whether the user is gazing at the AF frame based on the amount of overlap between the gaze pointer and the AF frame. That is, the system control unit 50 determines that the user is gazing at the AF frame when the amount of overlap between the gaze pointer and the AF frame exceeds a predetermined amount. For example, the predetermined amount may be half the gaze pointer. If the predetermined amount is half the gaze pointer, the system control unit 50 determines that the user is gazing at the AF frame if more than half of the gaze pointer overlaps the AF frame. The predetermined amount may be a fixed amount other than half the predetermined gaze pointer amount, or may be an amount that can be set by the user.

[0086] On the other hand, in S602, if the system control unit 50 determines that the user is not gazing at the AF frame, it repeats the process of S602 until it determines that the user is gazing at the AF frame.

[0087] As described above, in the third embodiment, the gaze return trigger condition in the gaze return trigger process is that the user is gazing at the AF frame (i.e., gazing at the AF frame). When the user gazes at the AF frame, the gaze-based subject selection function can be restored. This control reduces the execution of a specific function at a position unintended by the user, particularly at a position where a subject is not detected, and reduces missed shutter opportunities due to the need to select a subject again. Furthermore, in the third embodiment, in step S601, the EVF 29 displays the gaze pointer in a second predetermined color different from the first predetermined color, thereby intuitively informing the user that the specific function has not yet been executed at the gaze position.

[0088] In the above-described embodiments, the present invention has been described as being applied to an imaging device. However, the electronic devices to which the present invention can be applied are not limited to the imaging device described above. The present invention can also be applied to other electronic devices that have imaging means and zoom operation means and are capable of eye-gaze input. For example, the present invention can be applied to mobile terminals such as smartphones and tablet devices with camera functions that are capable of eye-gaze input, and digital video cameras that are capable of eye-gaze input. The present invention can also be applied to game consoles with camera functions that are capable of eye-gaze input, and wearable devices such as head-mounted displays that have an imaging unit. Furthermore, while the above-described embodiments are configured to achieve eye-gaze input functionality by combining the EVF 29 and the eye-gaze detection unit 160, the present embodiments can also be implemented by combining the eye-gaze detection unit 160 with a display device other than the EVF 29.

[0089] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and variations are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. [Explanation of symbols]

[0090] 50 System control section 100 Imaging device 150 Lens Unit 160 Gaze detection unit 7a Zoom ring 70 Operation section

Claims

1. A gaze detection means for detecting a user's gaze; a zoom operation means for performing a zoom operation; a selection means for selecting a subject from the image based on the position of the line of sight; a control means for controlling the selection of a subject based on the position of the line of sight without stopping the detection of the line of sight by the line of sight detection means in response to the start of the zoom operation by the zoom operation means; An electronic device comprising:

2. subject detection means for detecting a subject from an image; 2. The electronic device according to claim 1, wherein the selection means selects the object corresponding to the position of the line of sight detected by the line of sight detection means from among the objects detected by the object detection means.

3. The electronic device described in claim 1 or 2, characterized in that when a specific function is being executed on the selected subject, in response to the zoom operation being started by the zoom operation means, the control means stops the selection of the subject based on the gaze position without stopping gaze detection by the gaze detection means, and controls the specific function to continue to be executed on the subject that was selected based on the gaze position before the zoom operation was started.

4. 4. The electronic device according to claim 2, wherein, when the subject detected by the subject detection means is at the line of sight before the zoom operation is started and the subject detection means continues to detect the subject while the zoom operation is being performed, the control means controls the execution of the specific function on the subject selected based on the line of sight after the zoom operation is completed in response to a predetermined condition being satisfied.

5. 5. The electronic device according to claim 2, wherein the control means controls so as not to execute the specific function based on the line of sight while the zoom operation is being performed, regardless of whether or not the subject detected by the subject detection means is at the line of sight before the zoom operation is started.

6. 6. The electronic device according to claim 2, wherein, when the subject detected by the subject detection means is at the line of sight before the zoom operation is started and the subject becomes unable to be detected by the subject detection means while the zoom operation is being performed, the control means controls to stop execution of the specific function in response to the subject becoming unable to be detected.

7. 7. The electronic device according to claim 3, wherein the specific function is at least one of an autofocus function, an autoexposure function, an autowhite balance function, and a subject tracking function.

8. 5. The electronic device according to claim 4, wherein the predetermined condition is that a predetermined time has been measured by a timer.

9. The electronic device according to claim 8, characterized in that after the zoom operation is completed, the control means starts the timer and displays the gaze pointer indicating the gaze position in a second predetermined color different from the first predetermined color, and controls the gaze pointer to change from the second predetermined color to the first predetermined color when the timer has counted a predetermined time.

10. 5. The electronic device according to claim 4, wherein the predetermined condition is that a specific operating member is operated.

11. 5. The electronic device according to claim 4, wherein the predetermined condition is that the user is gazing at an AF frame set by the subject detection means.

12. 12. The electronic device according to claim 11, wherein the control means controls the electronic device so as to determine that the user is gazing at the AF frame when an amount of overlap between the gaze pointer indicating the position of the gaze and the AF frame exceeds a predetermined amount.

13. The electronic device described in claim 11 or 12, characterized in that the control means controls the display of a gaze pointer indicating the position of the gaze in a second predetermined color different from the first predetermined color after the zoom operation is completed, and when the user gazes at the AF frame, the gaze pointer is changed from the second predetermined color to the first predetermined color.

14. A method for controlling an electronic device, comprising: a gaze detection step of detecting a user's gaze; a zoom operation step of accepting a zoom operation; a selection step of selecting a subject from the image based on the position of the line of sight; a control step of performing control in response to the start of the zoom operation in the zoom operation step so as to stop the selection of the subject based on the position of the line of sight without stopping the detection of the line of sight in the line of sight detection step; 1. A method for controlling an electronic device, comprising:

15. A computer-readable storage medium having a program recorded thereon for causing a computer to execute a control method for an electronic device, The control method includes: a gaze detection step of detecting a user's gaze; a zoom operation step of accepting a zoom operation; a selection step of selecting a subject from the image based on the position of the line of sight; a control step of performing control in response to the start of the zoom operation in the zoom operation step so as to stop the selection of the subject based on the position of the line of sight without stopping the detection of the line of sight in the line of sight detection step; A storage medium comprising:

Images