Electronic devices, methods for controlling electronic devices, and computer-readable storage media

Abstract

This invention provides an electronic device, a control method for the electronic device, and a computer-readable storage medium. An electronic device includes: a gaze detection unit configured to receive gaze input from a user and detect the user's viewpoint position; an acquisition unit configured to acquire first distance information corresponding to the distance to the user's eyes; and a control unit configured to set a judgment time based on the first distance information, and to determine that a gaze has occurred if the viewpoint position remains unchanged for more than the judgment time.

Description

Technical Field

[0001] This invention relates to electronic devices, methods for controlling electronic devices, and computer-readable storage media. Background Technology

[0002] Various cameras have been proposed that detect the direction of the user's (photographer's) gaze, detect the area (position) viewed by the photographer in the viewfinder, and control various imaging functions such as autofocus control and auto exposure control. Japanese Patent Application Publication No. 2009-251658 discloses a technology for a display device in which the user's gaze is detected, and in which, if the user's gaze is fixed on a certain area for a period of time exceeding a predetermined threshold, it is determined that the user is looking at that area, thereby performing predetermined control.

[0003] A problem with conventional electronic devices for gaze detection is that the accuracy of gaze detection decreases when the positional relationship between the user's eyes and the device changes. One solution is to calibrate the device before use, ensuring accurate gaze input from the eye's position when the device is in use. However, if the user's eye position moves from the calibration point, incorrect gaze may be detected, potentially preventing the device from displaying the gaze position as intended and from functioning based on the gaze input. Summary of the Invention

[0004] In summary, the object of the present invention is to provide an electronic device that can perform eye-based input operations as intended by the user, even when the positional relationship between the user's eyes and the electronic device changes.

[0005] A first aspect of the present invention is an electronic device comprising: a gaze detection unit configured to receive gaze input from a user and detect the user's viewpoint position; an acquisition unit configured to acquire first distance information corresponding to the distance to the user's eyes; and a control unit configured to 1) set a judgment time based on the first distance information, and 2) determine that a gaze has occurred if the viewpoint position remains unchanged for more than the judgment time.

[0006] A second aspect of the present invention is: a control method for an electronic device, the electronic device including a gaze detection unit configured to receive gaze input from a user and detect the user's gaze position, the control method including: an acquisition step for acquiring first distance information corresponding to the distance to the user's eyes; and a control step for 1) setting a judgment time based on the first distance information, and 2) determining that a gaze has occurred if the gaze position remains unchanged for more than the judgment time.

[0007] A computer-readable storage medium storing a program for causing a computer to perform the steps of the control method described above.

[0008] Other features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0009] Figure 1A and 1B This is an external view of a digital camera according to the first embodiment;

[0010] Figure 2 This is a block diagram of a digital camera according to the first embodiment;

[0011] Figure 3 This is a flowchart depicting the process for selecting a ranging point according to the first embodiment;

[0012] Figure 4 This is a diagram illustrating the reason for changing the gaze determination time according to the first embodiment; and

[0013] Figure 5 This is a flowchart depicting the process for selecting a ranging point according to the second embodiment. Detailed Implementation

[0014] First Embodiment

[0015] <Exterior view of digital camera 100>: Preferred embodiments of the invention will be described with reference to the accompanying drawings. Figure 1A and 1B This is an external view of a digital camera 100, which is an example of an apparatus for applying the present invention. Figure 1A It is a front-view 3D image of the digital camera 100, and Figure 1B This is a 3D view of the back of the digital camera 100.

[0016] Display unit 28 is a display unit arranged on the back of digital camera 100, and displays images and various information. Touch panel 70a can detect touch operations performed on the display surface (touch operation surface) of display unit 28. External viewfinder display unit 43 is a display unit arranged on the upper surface of digital camera 100, and displays various settings of digital camera 100 such as shutter speed and aperture. Shutter button 61 is an operating member for indicating video recording. Mode selection switch 60 is an operating member for switching between various modes. Terminal cover 40 is a cover for protecting the connector (not shown) used to connect digital camera 100 to external devices.

[0017] The main electronic dial 71 is a rotary operating component, and settings such as shutter speed and aperture can be changed by rotating the main electronic dial 71. The power switch 72 is an operating component used to switch the power of the digital camera 100 on / off. The secondary electronic dial 73 is a rotary operating component, and for example, the selection box (cursor) can be moved and images switched by rotating the secondary electronic dial 73. The four-way key 74 is configured such that the upper, lower, left, and right parts of the key can be pressed respectively, thereby performing the corresponding actions. The setting button 75 is a push button and is mainly used to determine the selected item.

[0018] Video button 76 is used to start or stop shooting (recording) moving images. AE lock button 77 is a push button, and the exposure state can be fixed by pressing AE lock button 77 while the camera is in standby mode. Magnify button 78 is an operation button for turning the magnification mode on / off in the Live View Display (LV display) for switching video modes. If the main electronic dial 71 is operated after magnification mode is set to on, the Live View image (LV image) can be magnified or reduced. In playback mode, magnify button 78 is used to magnify or increase the magnification of the playback image. Playback button 79 is an operation button for switching between video mode and playback mode. If playback button 79 is pressed in video mode, the mode changes to playback mode, allowing the latest image from the recording medium 200 (described later) to be displayed on display unit 28. Menu button 81 is a button for indicating the display of a menu screen, and if menu button 81 is pressed, a menu screen allowing various settings is displayed on display unit 28. Users can intuitively make various settings using the menu screen, four-way key 74, and setting button 75 displayed on the display unit 28.

[0019] Communication terminal 10 is a communication terminal for communication between digital camera 100 and detachable lens unit 150 (described later). Eyepiece unit 16 is the eyepiece unit of eyepiece viewfinder (peep-type viewfinder), and the user can view the image displayed on the internal EVF 29 (described later) through eyepiece unit 16. Eye contact detection unit 57 is an eye contact detection sensor for detecting whether the user's (photographer's) eye is approaching eyepiece unit 16. Cover 202 is a cover for the slot storing recording medium 200 (described later). Grip unit 90 is a holding unit with a shape that allows the user to easily grip digital camera 100 with their right hand to take images. When the user holds digital camera 100 by gripping grip unit 90 with the little finger, ring finger, and middle finger of their right hand, shutter button 61 and main electronic dial 71 are in a position that can be operated by the right index finger. In this state, secondary electronic dial 73 is arranged in a position that can be operated by the right thumb. The thumb placement unit 91 (thumb standby position) is a grip member located on the back side of the digital camera 100, positioned so that the right thumb holding the grip unit 90 can be easily placed in a state where no operating member is being operated. The thumb placement unit 91 is made of rubber components or the like to increase holding force (grip feel).

[0020] <Block diagram of Digital Camera 100>: Figure 2 This is a block diagram depicting a configuration example of a digital camera 100. Lens unit 150 is a lens unit equipped with an interchangeable camera lens. Lens 103 typically consists of multiple lenses, but for simplicity... Figure 2 The image shows a lens. Communication terminal 6 is used for communication between lens unit 150 and digital camera 100, and communication terminal 10 is used for communication between digital camera 100 and lens unit 150. Lens unit 150 communicates with system control unit 50 via these communication terminals 6 and 10. Lens unit 150 then uses internal lens system control circuit 4 to control aperture 1 via aperture drive circuit 2. Lens unit 150 also uses lens system control circuit 4 to move lens 103 for focusing via AF drive circuit 3.

[0021] Shutter 101 is a focal plane shutter that can freely control the exposure time of the camera unit 22 based on the control of the system control unit 50.

[0022] The camera unit 22 is an imaging element composed of CCD or CMOS components, etc., to convert optical images into electrical signals. The camera unit 22 may include an imaging plane phase difference sensor that outputs defocus information to the system control unit 50.

[0023] The image processing unit 24 performs predetermined processing (pixel interpolation, resizing (e.g., reduction), or color conversion) on data from the A / D converter 23 or the memory control unit 15. The image processing unit 24 also performs predetermined calculations using the captured image data, and the system control unit 50 performs exposure control and range control based on the calculation results obtained by the image processing unit 24. This enables through-lens (TTL) type autofocus (AF) processing, automatic exposure (AE) processing, and pre-flash (EF) processing. Furthermore, the image processing unit 24 performs predetermined calculations using the captured data and performs TTL type automatic white balance (AWB) processing based on the obtained calculation results.

[0024] The memory control unit 15 controls the data transmission / reception between the A / D converter 23, the image processing unit 24, and the memory 32. Output data from the A / D converter 23 can be written to the memory 32 via the image processing unit 24 and the memory control unit 15. In some cases, output data from the A / D converter 23 can be written to the memory 32 via the memory control unit 15 without using the image processing unit 24. The memory 32 stores image data acquired by the camera unit 22 and converted into digital data by the A / D converter 23, and also stores image data to be displayed on the display unit 28 or EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and moving images and sounds of a predetermined duration.

[0025] Memory 32 is also a memory used for image display (video memory). Image data written to memory 32 for display is displayed on display unit 28 or EVF 29 via memory control unit 15. Display unit 28 and EVF 29 display on a monitor such as LCD or OLED according to signals from memory control unit 15. If data converted by A / D converter 23 and stored in memory 32 is sequentially transmitted and displayed on display unit 28 or EVF 29, live view display (LV display) can be performed. The image displayed on the live view display is called a "live view image (LV image)".

[0026] The gaze detection unit 160 detects the user's gaze in the eyepiece unit 16. The gaze detection unit 160 consists of a dichroic mirror 162, an imaging lens 163, a gaze detection sensor 164, a gaze detection circuit 165, and an infrared light-emitting diode 166. Since the system control unit 50 can perform predetermined processing based on the detection of the gaze, the gaze detection unit 160 can be considered part of the operation unit 70.

[0027] An infrared LED 166 is a light-emitting element used to detect the user's viewpoint position in the viewfinder image and emits infrared light towards the user's eyeball (eye) 161. The infrared light emitted from the infrared LED 166 is reflected by the eyeball (eye) 161, and the reflected infrared light reaches the dichroic mirror 162. The dichroic mirror 162 reflects only infrared light and allows visible light to pass through. The infrared reflected light, with its altered optical path, forms an image on the imaging surface of the gaze detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical component constituting the gaze detection optical system. The gaze detection sensor 164 is composed of an imaging device such as a CCD-type image sensor.

[0028] The gaze detection sensor 164 performs photoelectric conversion on the incident infrared reflected light and outputs the generated electrical signal to the gaze detection circuit 165. Based on the output signal of the gaze detection sensor 164, the gaze detection circuit 165 detects the user's gaze position according to the movement of the user's eyeball (eye) 161, and outputs the detected information to the system control unit 50 and the gaze determination unit 170.

[0029] The gaze input setting unit 167 enables / disables gaze detection performed by the gaze detection circuit 165 (gaze detection unit 160). The gaze input setting unit 167 also enables / disables gaze-based processing performed by the system control unit 50. For example, a user can freely set these enable / disable settings in the menu settings by operating the operation unit 70.

[0030] The gaze determination unit 170 determines that the user is looking at an area if the user's gaze is fixed on that area for a period exceeding a predetermined threshold, based on the detection information received from the gaze detection circuit 165. Therefore, that area can be considered the user's gaze location (gaze area). "The gaze is fixed on an area" means that the average position of the moving gaze is within that area until the predetermined time period has elapsed, and its variation (dispersion) is less than a predetermined value. The predetermined threshold can be freely changed by the system control unit 50. The gaze determination unit 170 does not need to be a separate block, and the system control unit 50 can perform the same functions as the gaze determination unit 170 based on the detection information received from the gaze detection circuit 165.

[0031] On the external viewfinder display unit 43, various camera settings such as shutter speed and aperture are displayed via the external viewfinder display unit drive circuit 44.

[0032] The non-volatile memory 56 is an electrically erasable and recordable memory such as a flash ROM. For example, constants and programs for the operating system control unit 50 are recorded in the non-volatile memory 56. Here, "program" refers to a program used to execute the various flowcharts described later in this embodiment.

[0033] The system control unit 50 is a control unit comprised of at least one processor or circuitry, and it controls the entire digital camera 100. The system control unit 50 implements the various processing steps of this embodiment (described later) by executing the programs stored in the non-volatile memory 56 described above. For example, the system memory 52 is RAM, and the system control unit 50 expands the constants and variables used by the operating system control unit 50, as well as the programs read from the non-volatile memory 56, in the system memory 52. ​​The system control unit 50 also controls the display by controlling the memory 32 and the display unit 28, etc.

[0034] System timer 53 is a clock unit used to measure the time used for various controls and the time of the internal clock.

[0035] The power control unit 80 comprises a battery detection circuit, a DC-DC converter, and a switching circuit for switching which blocks are to be powered. It detects, for example, whether a battery is installed, the type of battery, and the remaining battery power. Based on this detection result and instructions from the system control unit 50, the power control unit 80 controls the DC-DC converter and supplies the required voltage to each unit, including the recording medium 200, during the required time period. The power supply unit 30 comprises a primary battery (such as an alkaline battery or lithium battery), a secondary battery (such as a NiCd battery, NiMH battery, or Li battery), and an AC adapter.

[0036] Recording medium I / F 18 is an interface with recording medium 200 (e.g., memory card, hard disk). Recording medium 200 is a recording medium, such as a memory card, used to record captured images, and is composed of semiconductor memory or a hard disk, etc.

[0037] Communication unit 54 transmits / receives video and audio signals to / from external devices connected wirelessly or via cable. Communication unit 54 can also connect to a wireless local area network (LAN) and the Internet. Communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. Communication unit 54 can transmit images captured by camera unit 22 (including LV images) and images recorded on recording medium 200, and can receive image data and various other information from external devices.

[0038] The posture detection unit 55 detects the posture of the digital camera 100 relative to the direction of gravity. Based on the posture detected by the posture detection unit 55, it can be determined whether the image captured by the imaging unit 22 was captured by the horizontally held digital camera 100 or by the vertically held digital camera 100. The system control unit 50 can append orientation information corresponding to the posture detected by the posture detection unit 55 to the image file of the image captured by the imaging unit 22, or it can rotate and record the image. For the posture detection unit 55, an accelerometer or a gyroscope sensor can be used. The accelerometer or gyroscope sensor of the posture detection unit 55 can also be used to detect the movement of the digital camera 100 (e.g., pan, tilt, tilt, stationary).

[0039] The eye-to-eye detection unit 57 is an eye-to-eye detection sensor used to detect the approach (eye-to-eye) and departure (away from eye) of the eye (object) 161 to / from the eyepiece unit 16 of the viewfinder 17 (hereinafter referred to as "viewfinder"). The system control unit 50 switches between display unit 28 and EVF 29 between display (display state) and non-display (non-display state) based on the state detected by the eye-to-eye detection unit 57. Specifically, when the digital camera 100 is at least in video standby mode and the display destination switching setting is in automatic switching mode, in the non-eye-to-eye state, the display destination is set to display unit 28, causing the display of display unit 28 to be on and the EVF 29 to be off. On the other hand, in the eye-to-eye state, the display destination is set to EVF 29, causing the display of EVF 29 to be on and the display unit 28 to be off. For example, an infrared proximity sensor can be used for the eye-to-eye detection unit 57 to detect the approach of an object to the eyepiece unit 16 of the viewfinder 17, which includes the EVF 29. When an object approaches, infrared light emitted from the light-emitting unit (not shown) of the eye-close detection unit 57 is reflected by the object and received by the light-receiving unit (not shown) of the infrared proximity sensor. The distance from the object to the eyepiece unit 16 (eye-close distance) can also be determined by the amount of infrared light received. In this way, the eye-close detection unit 57 performs eye-close detection to detect the approach distance of the object to the eyepiece unit 16. It is assumed that eye-close is detected when an object is detected approaching the eyepiece unit 16 at a predetermined distance or less after a non-eye-close state (non-proximity state). It is also assumed that eye-away is detected when an approaching object is detected to have moved at least a predetermined distance away from the eye-close state (eye-proximity state). For example, a threshold for detecting eye-close and a threshold for detecting eye-away can be distinguished by hysteresis. After eye-close is detected, it is assumed that the eye-close state continues until eye-away is detected. Then, after eye-away is detected, it is assumed that the non-eye-close state continues until eye-close is detected. It should be noted that the infrared proximity sensor is only an example, and other types of sensors can be used for the eye-contact detection unit 57, as long as the sensor can detect the approach of the eye or the approach of an object that can be considered as eye-contact.

[0040] The system control unit 50 can detect the following operations performed on the eyepiece unit 16 or the state of the eyepiece unit 16 by controlling the gaze determination unit 170 or the eye contact detection unit 57: 1) A line of sight not directed at the eyepiece unit 16 is turned towards the eyepiece unit 16 (i.e., the start of gaze input). 2) A line of sight is being input to the eyepiece unit 16. 3) The eyepiece unit 16 is being gazed at. 4) A line of sight directed at the eyepiece unit 16 is turned away (i.e., the end of gaze input). 5) No gaze is input to the eyepiece unit 16.

[0041] These operations, statuses, and the position (direction) of the line of sight relative to the eyepiece unit 16 are notified to the system control unit 50 via the internal bus, and the system control unit 50 determines the type of operation (line of sight operation) performed on the eyepiece unit 16 based on the notified information.

[0042] The operation unit 70 is an input unit that receives operations performed by the user (user operations) and is used to input various operation commands to the system control unit 50. For example... Figure 2 As shown, the operation unit 70 includes a mode selection switch 60, a shutter button 61, a power switch 72, and a touch panel 70a. The operation unit 70 also includes other operation components 70b, such as a main electronic dial 71, a secondary electronic dial 73, a four-way key 74, a setting button 75, a video button 76, an AE lock button 77, a zoom button 78, a playback button 79, and a menu button 81.

[0043] The mode selection switch 60 switches the operating mode of the system control unit 50 to still image shooting mode, moving image shooting mode, or playback mode, etc. The still image shooting mode includes the following modes: automatic recording mode, automatic scene detection mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). It also includes various scene modes and custom modes for setting up recording for different shooting scenes. Users can directly select any of these modes using the mode selection switch 60. Alternatively, users can first use the mode selection switch 60 to select a list of shooting modes, and then use other operating components to select any of the multiple modes displayed in the list. Similarly, the moving image shooting mode can include multiple modes.

[0044] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. During operation of the shutter button 61 (i.e., in a half-pressed state (video recording preparation command)), the first shutter switch 62 is activated, generating a first shutter switch signal SW1. Through the first shutter switch signal SW1, the system control unit 50 begins video recording preparation operations, such as autofocus (AF) processing, auto exposure (AE) processing, auto white balance (AWB) processing, and pre-flash (EF) processing. When the operation of the shutter button 61 is complete (i.e., in a fully pressed state (video recording command)), the second shutter switch 64 is activated, generating a second shutter switch signal SW2. Through the second shutter switch signal SW2, the system control unit 50 begins a series of video recording operations, from reading signals from the imaging unit 22 to writing the captured image as an image file to the recording medium 200.

[0045] The touch panel 70a and the display unit 28 can be integrated. For example, the touch panel 70a is configured such that its light transmittance does not obstruct the display on the display unit 28, and is superimposed on the display surface of the display unit 28. Then, the input coordinates on the touch panel 70a correspond to the display coordinates on the display surface of the display unit 28. Thus, a graphical user interface (GUI) can be provided that allows the user to directly operate the screen displayed on the display unit 28. The system control unit 50 can detect the following operations on the touch panel 70a or the state of the touch panel 70a.

[0046] 1) No finger or stylus is touching touch panel 70a, i.e., touch begins (hereinafter referred to as Touch-Down). 2) A finger or stylus is touching touch panel 70a (hereinafter referred to as Touch-On). 3) A finger or stylus is moving while touching touch panel 70a (hereinafter referred to as Touch-Move). 4) The finger or stylus touching touch panel 70a is released from touch panel 70a, i.e., touch ends (hereinafter referred to as Touch-Up). 5) No object is touching touch panel 70a (hereinafter referred to as Touch-Off).

[0047] If a touch is detected, touch persistence is also detected. Touch persistence is generally detected unless touch cessation is detected after the touch. If touch movement is also detected, touch persistence is also detected. Even if touch persistence is detected, touch movement will not be detected unless the touch location is moving. If all finger or stylus touches are detected to have stopped, no touch is detected.

[0048] The system control unit 50 is notified of these operations, states, and the coordinates of the position of the finger or pen touching the touch panel 70a via an internal bus. The system control unit 50 then determines the type of operation (touch operation) performed on the touch panel 70a based on the notified information. For touch movement, the system control unit 50 can also determine the direction of movement of the finger or pen on the touch panel 70a based on changes in position coordinates, specifically for the vertical and horizontal components. If a touch movement of at least a predetermined distance is detected, the system control unit 50 determines that a swipe operation has been performed. An operation in which the finger moves quickly a certain distance on the touch panel 70a while in a touching state and then releases the finger is called a "flick." In other words, a flick is an operation of quickly moving and releasing the finger on the touch panel 70a. If a touch movement of at least a predetermined distance at a predetermined speed or faster is detected, and then the touch stops, the system control unit 50 determines that a flick has been performed (determined to have occurred after a swipe operation). Furthermore, a touch operation that simultaneously touches multiple points (e.g., two points) and moves these touch locations closer together is called a "pinch-in," while a touch operation that moves these touch locations further apart is called a "pinch-out." Pinch-in and pinch-out are collectively referred to as a pinch-out operation (or simply "pinch-out"). For the touch panel 70a, various types of touch panels can be used, such as resistive film type, capacitive type, surface acoustic wave type, infrared type, electromagnetic induction type, image recognition type, and optical sensor type. There are types that detect touch when the touch panel is actually in contact with it, and types that detect touch when a finger or pen approaches the touch panel; either type can be used.

[0049] <Processing for selecting distance measurement points>: Refer to... Figure 3 The process of selecting a ranging point according to the first embodiment is described, which uses gaze determination time in the digital camera 100 based on eye contact distance. In the first embodiment, it is assumed that "distance" refers to the interval (distance) between two objects (the spatial length between two objects).

[0050] Figure 3 This is a flowchart depicting the process of selecting a ranging point according to the first embodiment. In the first embodiment, in the digital camera 100, the gaze determination time Tth, which serves as a threshold (predetermined threshold) for determining whether a gaze has occurred, is switched based on the distance between the user's eye 161 and the gaze detection unit 160. If the digital camera 100 according to the first embodiment is used, the user can appropriately select a ranging point while peering through the EVF 29, even if the position of the user's eye may move. Figure 3The processes in the flowchart are implemented by the system control unit 50. The system control unit 50 expands the programs stored in the non-volatile memory 56 into the system memory 52 and executes the programs to control the various functional units. When the power switch 72 is turned on and the digital camera 100 is started, the process begins... Figure 3 The flowchart.

[0051] In S301, the system control unit 50 controls the eye contact detection unit 57 and measures (acquires) the distance Leye between the user's eye 161 and the gaze detection unit 160. The distance Leye between the user's eye 161 and the gaze detection unit 160 can be the optical distance (distance information) between the user's eye 161 and the gaze detection sensor. In the first embodiment, the gaze detection unit 160 and the eyepiece unit 16 are arranged adjacent to each other, so the distance Leye can be considered as the distance between the user's eye 161 and the eyepiece unit 16 (eye contact detection unit 57).

[0052] In S302, the system control unit 50 determines whether the distance Leye measured in S301 is less than a first threshold Lth1. If the distance Leye is less than the first threshold Lth1, the system control unit 50 determines that the user's eye is close to the eyepiece unit 16 (viewfinder). On the other hand, if the distance Leye is at least the first threshold Lth1, the system control unit 50 determines that the state is an in-eye state. If the distance Leye is less than the first threshold Lth1, the process proceeds to S303; otherwise, the process returns to S301. In other words, steps S301 and S302 are repeated until the state where the user's eye is detected to be close to the eyepiece unit 16 is reached.

[0053] In S303, the system control unit 50 controls the gaze detection unit 160 and detects the user's viewpoint position (the position viewed in the EVF 29). In this case, the viewpoint position does not need to be detected in pixels in the EVF 29, but can be detected as a block dividing the display area in the EVF 29, or as a subject. The system control unit 50 stores the detected viewpoint position in the memory 32 to correspond to the detection time.

[0054] In S304, the system control unit 50 determines whether the viewpoint position detected in S303 is different from the previous viewpoint position detected in S303 stored in the memory 32. Since this is the first time the process in S303 has been performed, the memory 32 does not store information related to the previously detected viewpoint position; therefore, the system control unit 50 determines that the viewpoint position is different between the previous and current positions. If the viewpoint position is different between the current and previous positions, the process proceeds to S305; otherwise, the process proceeds to S306. Here, the system control unit 50 can determine from the multiple viewpoint positions stored in the memory 302 that the previous viewpoint position is a viewpoint position associated with a time closest to the time associated with the current viewpoint position.

[0055] When the viewpoint position has high resolution (e.g., when the viewpoint position is obtained in pixels), if the user's viewpoint position shifts slightly, it is determined that the viewpoint position has changed relative to the previous viewpoint position. Therefore, when the viewpoint position has high resolution, if the current viewpoint position is within a predetermined range of the previous viewpoint position, it can be considered that the viewpoint position is the same as the previous viewpoint position.

[0056] In S305, the system control unit 50 resets the gaze timer Teye, which measures the duration for which the viewpoint position remains unchanged (setting the gaze timer Teye to 0).

[0057] In S306, the system control unit 50 determines whether the distance Leye measured in S301 is greater than the second threshold Lth2. Here, the second threshold Lth2 is a value smaller than the first threshold Lth1. If the distance Leye is greater than the second threshold Lth2, the process proceeds to S307; otherwise, the process proceeds to S308.

[0058] In S307, the system control unit 50 sets the first judgment time T1 as the gaze judgment time Tth.

[0059] In S308, the system control unit 50 determines whether the distance Leye measured in S301 is greater than the third threshold Lth3. Here, the third threshold Lth3 is a value smaller than the second threshold Lth2. In other words, Lth1 > Lth2 > Lth3 holds true. An example is Lth1 = 10mm, Lth2 = 5mm, and Lth3 = 3mm. If the distance Leye is greater than the third threshold Lth3, the process proceeds to S309; ​​otherwise, the process proceeds to S310.

[0060] In S309, the system control unit 50 sets the second judgment time T2 to the gaze judgment time Tth. In S310, the system control unit 50 sets the third judgment time T3 to the gaze judgment time Tth. Here, the second judgment time T2 is shorter than the first judgment time T1. Furthermore, the third judgment time T3 is shorter than the second judgment time T2. In other words, T1>T2>T3 holds true.

[0061] In S311, the system control unit 50 controls the gaze determination unit 170 and determines whether the value of the gaze timer Teye is greater than the gaze determination time Tth. In other words, the system control unit 50 (gaze determination unit 170) determines whether the time during which the viewpoint position remains unchanged is longer than the gaze determination time Tth, and if it is longer than the gaze determination time Tth, it determines that the viewpoint position is being gazed upon. If the value of the gaze timer Teye is greater than the gaze determination time Tth, the process proceeds to S312; otherwise, the process returns to S301.

[0062] In S312, the system control unit 50 determines (selects) a ranging point at a location corresponding to the viewpoint position (gaze position). In the first embodiment, the system control unit 50 determines the ranging point in S312, but can select the icon displayed at the viewpoint position (gaze position) and can zoom in or out on the image centered on the viewpoint position. Furthermore, the system control unit 50 can control the EVF 29 to display the gaze position (display an item indicating the gaze position). In other words, in S312, the system control unit 50 can perform any processing as long as it is in response to the occurrence of a gaze.

[0063] (The reason for judging time based on changes in gaze distance): will refer to Figure 4 This explains why the gaze determination time Tth is changed based on the distance Leye. Figure 4 It is a graph depicting the time-varying distance Leye between the user's eye 161 and the gaze detection unit 160.

[0064] The case where the distance Leye is greater than the second threshold Lth2 (where the distance Leye is within range C) indicates that the user's eye has moved significantly away from the viewfinder due to camera shake, etc. For example, when the distance between the user's eye 161 and the gaze detection unit 160 is long, even a slight movement of the user's eye 161 significantly moves the viewpoint position to be detected by the gaze detection unit 160. Specifically, the viewpoint position to be detected may be significantly moved by involuntary eye movements during fixation, which is an unintentional eye movement by the user. This means that when the distance Leye is within range C, the detection accuracy of the gaze detection unit 160 used to detect the viewpoint position is not very good. Therefore, the system control unit 50 sets the gaze judgment time to a relatively long time to ensure that the detection accuracy of the gaze detection unit 160 is stable, so as to detect the viewpoint position under good conditions. For example, when the user's eye has moved away from the viewfinder due to camera shake, since the period of camera shake is usually about 1 to 10 Hz, if the first judgment time T1 is about 100 ms to 1 s, it can be ensured that the accurate viewpoint position is acquired. If the poor detection accuracy used to detect the viewpoint position continues in a stable state and is not caused by periodic changes such as camera shake, setting the gaze determination time to be relatively long can prevent rapid determination of the gaze position that would lead to errors, thereby preventing unintentional user actions.

[0065] It should be noted that when the distance Leye is greater than the second threshold Lth2 (when the distance Leye is within range C), in S307, the system control unit 50 can control the EVF 29 to display a warning to the user. This warning display could be a warning to notify of low accuracy in gaze detection, or a warning to indicate that the eye is very close to the viewfinder. Furthermore, when the distance Leye is greater than the second threshold Lth2, in S307, the system control unit 50 can disallow (disable) receiving gaze input from the gaze detection unit 160 without setting the gaze determination time Tth.

[0066] The case where the distance to Leye is not greater than the second threshold Lth2 but greater than the third threshold Lth3 (the case where the distance to Leye is within range B) indicates that the user's eye has moved slightly away from the viewfinder due to camera shake, etc. In this way, when the distance to Leye is within range B, the detection accuracy for detecting the viewpoint position decreases due to camera shake, etc., but the detection accuracy of the gaze detection unit 160 for detecting the viewpoint position is higher than when the distance to Leye is within range C. Therefore, the system control unit 50 sets the second judgment time T2, which is shorter than the first judgment time T1, as the gaze judgment time Tth.

[0067] When the distance Leye is not greater than the third threshold Lth3 (when the distance Leye is within range A), it indicates that the user's eye is reliably approaching (fixed) the viewfinder. If the distance between the user's eye 161 and the gaze detection unit 160 is short at this time, the position of the viewpoint to be detected will not change significantly even if the user's eye 161 moves. For example, the viewpoint position will not change much due to involuntary eye movements during fixation. Therefore, when the distance Leye is within range A, the gaze detection unit 160 has good detection accuracy for detecting the viewpoint position. Therefore, the system control unit 50 sets the third judgment time T3, which is shorter than the second judgment time T2, as the gaze judgment time Tth.

[0068] As described above, the detection accuracy of the viewpoint position varies depending on the distance between the gaze detection unit 160 and the user's eye 161. Therefore, if the gaze judgment time is set according to the distance, processing such as distance point selection can be achieved as intended by the user.

[0069] In the first embodiment, the system control unit 50 changes the gaze determination time regardless of the operating mode, but it can switch (select) whether to change the gaze determination time based on the distance between the gaze detection unit 160 and the user's eyes 161, depending on the operating mode. For example, when capturing still images (still image shooting mode), the ranging point must be selected quickly, so the system control unit 50 can set the gaze determination time to a short value (e.g., a third determination time T3) regardless of the distance between the gaze detection unit 160 and the user's eyes 161. On the other hand, when capturing moving images (moving image shooting mode), the system control unit 50 controls the gaze determination time to change according to the distance between the gaze detection unit 160 and the user's eyes 161. Furthermore, in both the moving image shooting mode and the still image shooting mode, the gaze determination time can be determined to be short regardless of the distance between the gaze detection unit 160 and the user's eyes 161. In this case, in the editing mode of editing captured images, real-time operation is not required, and the gaze determination time can be changed according to the distance between the gaze detection unit 160 and the user's eyes 161. Furthermore, the system control unit 50 can set the gaze determination time based on the distance between the gaze detection unit 160 and the user's eyes 161 in both still image capture mode and moving image capture mode. In this case, if the distance between the gaze detection unit 160 and the user's eyes 161 is the same in both modes, the system control unit 50 can set the gaze determination time in still image capture mode to be shorter than that in moving image capture mode.

[0070] In the detection of gaze position (viewpoint position) by the gaze detection unit 160, if information related to the reliability (accuracy) of the viewpoint position detection can be obtained based on past detection results, the gaze determination time can be set based on that result. Specifically, the gaze determination time can be set to be shorter as the reliability of past viewpoint position detection is higher, and longer as the reliability is lower.

[0071] Furthermore, in the first embodiment, the gaze determination time varies based on the distance (numerical value; distance information) between the gaze detection unit 160 and the user's eyes 161. However, regarding the gaze determination time, an arithmetic expression based on the distance between the gaze detection unit 160 and the user's eyes 161 can be used to determine the gaze determination time. For example, the gaze determination time is set to increase as the distance between the gaze detection unit 160 and the user's eyes 161 increases. Specifically, the gaze determination time can be set such that the distance between the gaze detection unit 160 and the user's eyes 161 is proportional to the length of the gaze determination time. Furthermore, in the first embodiment, the change in distance between the user's eyes 161 and the gaze detection unit 160 per unit time (…) can be used. Figure 4 The tilt of the image is used to represent the distance (numerical value; distance information) between the user's eye 161 and the gaze detection unit 160. In other words, the system control unit 50 can acquire the amount of change in the distance between the user's eye 161 and the gaze detection unit 160 per unit time, and the gaze determination time is set to increase as the amount of change increases. This is because as the amount of change in the distance between the user's eye 161 and the gaze detection unit 160 with respect to time increases, the positional relationship between the user's eye 161 and the gaze detection unit 160 changes significantly, that is, the accuracy of acquiring the viewpoint position decreases.

[0072] Furthermore, compared to other timing methods, the detection accuracy of the gaze detection unit 160, which detects the gaze, may deteriorate immediately after the user's eye approaches the eyepiece unit 16. Therefore, the system control unit 50 can adjust the gaze determination time based on the duration of eye contact. For example, in S302, the system control unit 50 can determine not only whether the user's eye is close to the eyepiece unit 16 (viewfinder), but also the duration of eye contact. When determining the duration of eye contact, the system control unit 50 sets the gaze determination time to be longer as the duration of eye contact is shorter. If the duration of eye contact is shorter than a predetermined time, the system control unit 50 can set a first determination time T1 or a time longer than T1 as the gaze determination time, regardless of the distance between the eye and the gaze detection unit 160.

[0073] Second Embodiment

[0074] Reference Figure 5 The process of selecting a ranging point by a digital camera 100 according to the second embodiment is described. The configuration of the digital camera 100 according to the second embodiment is the same as that of the digital camera 100 according to the first embodiment, therefore its description is omitted.

[0075] Most electronic devices (gaze input devices) that can be used as gaze detection unit 160 require calibration (correcting the parameters used by gaze detection unit 160 for gaze detection; correcting the correspondence between the user's eye direction and viewpoint position). One reason for the need for calibration is that if the distance between the gaze input device and the eye of the object being detected changes, a difference will be generated in the viewpoint position detection result. Therefore, calibration is performed in advance when using digital camera 100 to ensure the accuracy of viewpoint position (gaze) detection. However, in the case of digital camera 100, the positional relationship between digital camera 100 and user is prone to change, and in some cases, it is difficult to maintain the same positional relationship even if calibration is performed in advance. For example, when using digital camera 100 in a shooting environment where camera shake or user movement is likely to occur, it is difficult to maintain the same positional relationship between user and digital camera.

[0076] Therefore, in the second embodiment, a digital camera 100 will be described that not only uses the results of calibration but also considers changes in the positional relationship between the user's eyes 161 and the gaze detection unit 160. According to the second embodiment, the digital camera 100 switches the gaze determination time Tth based on the distance between the user's eyes 161 and the gaze detection unit 160 at the timing of calibration performed on the gaze detection unit 160.

[0077] Figure 5 This is a flowchart depicting the process of selecting a ranging point according to the second embodiment. Figure 5 Each process in the flowchart is implemented by the system control unit 50. The system control unit 50 expands the programs stored in the non-volatile memory 56 into the system memory 52 and executes the programs to control the various functional units. When the digital camera 100 is started and the system control unit 50 completes the calibration, it begins... Figure 5 The flowchart is as follows. When calibration is performed by controlling the eye-sensing detection unit 57, the system control unit 50 pre-measures (acquires) the distance Lcal between the user's eye 161 and the gaze detection unit 160. Then, the system control unit 50 stores the distance Lcal acquired during calibration into the memory 32. Figure 5 Processing steps S301 to S305 and S307 to S312 in the first embodiment are the same as those in the first embodiment. Figure 3 The processing steps in the flowchart are the same, so their description is omitted.

[0078] In S506, the system control unit 50 determines whether the absolute difference Labs, which is the absolute value of the difference between the distance Leye and the distance Lcal measured in S301 (|Leye-Lcal|), is greater than the second threshold Lth2. If the absolute difference Labs is greater than the second threshold Lth2, the process proceeds to S307; otherwise, the process proceeds to S508.

[0079] In S508, the system control unit 50 determines whether the absolute difference Labs is greater than the third threshold Lth3. If the absolute difference Labs is greater than the third threshold Lth3, the process proceeds to S309; ​​otherwise, the process proceeds to S310.

[0080] If the absolute difference Labs is greater than the second threshold Lth2, it indicates that the positional relationship between the user's eye 161 and the gaze detection unit 160 deviates significantly from the positional relationship during the calibration time. Therefore, the detection accuracy of the gaze detection unit 160 in detecting the viewpoint position may be poor. Thus, in S307, the system control unit 50 sets the gaze determination time Tth to the first determination time T1, which is the largest value.

[0081] The case where the absolute difference Labs is not greater than the second threshold Lth2 but greater than the third threshold Lth3 indicates a slight deviation in the positional relationship between the user's eye 161 and the gaze detection unit 160 from the positional relationship in the calibration time. In this case, the detection accuracy of the gaze detection unit 160 for detecting the viewpoint position may not be good. Therefore, in S309, the system control unit 50 sets the gaze determination time Tth to a second determination time T2, which is a smaller value than the first determination time T1.

[0082] The fact that the absolute difference Labs is not greater than the third threshold Lth3 indicates that the positional relationship between the eye and the digital camera 100 during calibration can be maintained, so the gaze detection unit 160 can accurately detect the gaze position. Therefore, in S310, the system control unit 50 sets the gaze determination time Tth to the third determination time T3 as the shortest determination time.

[0083] In this way, the gaze determination time varies depending on the distance between the user's eye 161 and the gaze detection unit 160 during calibration and when using the digital camera 100. This allows the ranging point to be selected based on the gaze input, reflecting calibration and as intended by the user.

[0084] In the second embodiment, the gaze determination time is changed based on whether the difference between the distance between the user's eye 161 and the gaze detection unit 160 during calibration and the distance when using the digital camera 100 is greater than or less than a threshold. However, instead of using this change method, the gaze determination time can be determined by an arithmetic expression based on distance information. For example, the larger the absolute value of the difference between distance Leye and distance Lcal, the longer the gaze determination time Tth can be set.

[0085] To set the gaze determination time Tth, it is not always necessary to use the absolute value (numerical value) of the difference between distance Leye and distance Lcal as described in the second embodiment. To set the gaze determination time Tth, the ratio (percentage) of distance Leye to distance Lcal, i.e., Leye / Lcal, can be used instead of this absolute value.

[0086] Although the invention has been described with reference to preferred embodiments, the invention is not limited to these specific embodiments and includes various modes without departing from the spirit of the invention. Furthermore, the embodiments described above are merely examples of the invention, and embodiments may be combined where necessary.

[0087] For example, eye-contact detection by eye-contact detection unit 57 can be performed using other ranging methods such as lasers, instead of infrared proximity sensors. Furthermore, in the above description, the distance between eye-contact detection unit 160 and the user's eyes 161 is obtained by eye-contact detection unit 57; however, an acquisition unit could be arranged to obtain the distance by comparing two images captured at two different locations. Moreover, the description of the above embodiments is based on the assumption of using a camera; however, digital camera 100 is not limited to a camera but can be an electronic device including image capture functionality. For example, the present invention is applicable to electronic devices used in a head-mounted state, such as head-mounted displays (HMDs), because the distance between the eye-contact detection unit and the eyes determined in the mounted state varies depending on the user or the mounting environment (wearing status of glasses, model, individual differences).

[0088] According to the present invention, even if the positional relationship between the user's eyes and the electronic device changes, operation based on gaze input can be performed as intended by the user.

[0089] Other embodiments

[0090] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0091] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An electronic apparatus comprising: a line-of-sight detection unit configured to receive a line-of-sight input from a user of the electronic apparatus and to detect a point-of-gaze position of the user; an acquisition unit configured to acquire first distance information corresponding to a distance to an eye of the user; and a control unit configured to set a judgment time based on the first distance information, and to judge that a gaze has occurred in a case where it is judged that the point-of-gaze position has not changed for more than the judgment time, wherein, in a case where a value based on the first distance information is greater than a predetermined threshold value, the control unit sets the judgment time to be longer than in a case where the value is not greater than the predetermined threshold value. The control unit is further configured to set the judgment time based on an amount of change per unit time of the first distance information. 2.The electronic device of claim 1, wherein, The control unit is further configured to set the judgment time based on the first distance information at a point in time at which the control unit sets the judgment time and second distance information corresponding to a distance to the eye of the user that is acquired in advance, 3.The electronic device of claim 1, wherein wherein, the control unit is further configured to perform calibration of the line-of-sight detection unit before setting the judgment time, and the second distance information is information corresponding to a distance to the eye of the user at a point in time at which the control unit performed the calibration. The control unit is further configured to set the judgment time based on an absolute value of a difference between a distance corresponding to the first distance information and a distance corresponding to the second distance information. 4.The electronic device of claim 3, wherein, The control unit is further configured to set the judgment time based on a ratio of a distance corresponding to the first distance information and a distance corresponding to the second distance information. 5.The electronic device of claim 3, wherein The control unit is further configured to set the judgment time based on a reliability of past detection of a point-of-gaze position by the line-of-sight detection unit. 6.The electronic device of claim 1, wherein 7.The electronic apparatus according to claim 1, further comprising a viewfinder to which an eye of the user approaches, wherein the control unit is further configured to acquire a time for which the eye of the user continuously approaches the viewfinder, and to set the judgment time based on the acquired time. In a case where a value based on the first distance information is greater than a predetermined threshold value, the control unit controls the line-of-sight detection unit such that the line-of-sight input from the user is not received. 8.The electronic device of claim 1, wherein, In a case where a value based on the first distance information is greater than a predetermined threshold value, the control unit controls a display to display a warning to the user. 9.The electronic device of claim 1, wherein In a case where it is judged that a gaze has occurred, the control unit further performs processing corresponding to a gaze position, which is the point-of-gaze position at a point in time at which it is judged that a gaze has occurred. 10.The electronic device of claim 1, wherein In a case where it is judged that a gaze has occurred, the control unit sets a ranging point to a position corresponding to the gaze position. 11.The electronic device of claim 10, wherein In a case where it is judged that a gaze has occurred, the control unit controls a display to display the gaze position. 12.The electronic device of claim 10, wherein, 13.The electronic apparatus according to claim 1, wherein the electronic apparatus includes a first mode and a second mode, ​ In the first mode, the control unit sets the determination time based on the first distance information, and In the second mode, the control unit sets the determination time without relying on the first distance information.

14. The electronic apparatus according to claim 13, wherein the first mode is a mode of capturing a moving image, and the second mode is a mode of capturing a still image.

15. The electronic device according to any one of claims 1 to 14, wherein The distance to the user's eye is a distance between the user's eye and the gaze detection unit, or a distance between the user's eye and a viewfinder to which the user's eye approaches.

16. A control method of an electronic apparatus including a gaze detection unit configured to receive a gaze input from a user of the electronic apparatus and detect a gaze point position of the user, the control method comprising: an acquisition step of acquiring first distance information corresponding to a distance to the user's eye; and a control step of setting a determination time based on the first distance information, and determining that a gaze has occurred in a case where it is determined that the gaze point position has not changed continuously for more than the determination time, wherein, in a case where a value based on the first distance information is larger than a predetermined threshold value, the determination time is set to be longer in the control step than in a case where the value is not larger than the predetermined threshold value.

17. A computer-readable storage medium storing a program for causing a computer to execute the steps of the control method according to claim 16.

18. A computer program product including a program for causing a computer to execute the steps of the control method according to claim 16. ​

Images