Electronic device, control method, and computer-readable medium

Abstract

The present invention relates to an electronic device, a control method, and a computer-readable medium. The electronic device includes: a sight line input unit configured to receive sight line input using the sight line of a user viewing a display unit; an acquisition unit configured to acquire an image captured by a predetermined camera control; a display control unit configured to display the acquired image on the display unit; a first operation unit configured to receive a first operation input from the user; a second operation unit configured to receive a second operation input from the user; and a control unit configured to: when the image is displayed, perform setting of the predetermined camera control based on the sight line position when the first operation input is performed, and perform image processing on the displayed image based on the sight line position when the second operation input is performed.

Description

Technical Field

[0001] The present invention relates to an electronic device, and more particularly to an electronic device capable of receiving line-of-sight input. Background Art

[0002] Electronic devices that are operated by a user using their line of sight (hereinafter referred to as line of sight input) are known. Line of sight input is particularly effective when a user wants to quickly instruct an operation of an electronic device such as a digital camera or a game console.

[0003] Japanese Patent Laid-Open No. 2014-197109 discloses judging a user's attention area on which a user is paying attention, and based on the judgment result, performing control to distinguish the brightness of the user's attention area from the brightness of a non-attention area, which is an area other than the user's attention area, in a game image.

[0004] As a function (technology) related to cameras, a function has been proposed that continuously performs focus control (control that receives line-of-sight input and adjusts the focus at the AF frame position according to the line-of-sight position (the position to which the line-of-sight input is directed)) by line-of-sight input. In this function, the user must look at the subject to be focused on in the displayed live view image without looking away from the subject to be focused (in-focus).

[0005] Furthermore, by using the technology disclosed in Japanese Patent Laid-Open No. 2014-197109, it is possible to control the brightness of the area of ​​interest of the line of sight input in the live view image (brightness control using the line of sight input).

[0006] However, if a function of brightness control using line-of-sight input is added to an electronic device that constantly performs focus control using line-of-sight input, focus control, etc. may not be performed appropriately (may not be performed with good operability). For example, it is assumed here that the user is viewing a subject B different from the subject A to be focused on in the live view image in order to control the brightness of subject B. In this case, the brightness of subject B can be controlled as the user intended, but the focus target is switched from subject A to subject B, contrary to the user's intention. In addition, in some cases, the user may check an area other than the subject to be focused on to confirm the framing (composition). Also, in the case of such a check, the focus target may be switched, contrary to the user's intention. Summary of the Invention

[0007] The present invention provides an electronic device that can appropriately (with good operability) perform a plurality of controls (processing operations) such as focus control and brightness control using line-of-sight input.

[0008] According to the present invention, the electronic device includes: an electronic device, comprising: a line of sight input unit, which is configured to receive a line of sight input performed by the line of sight of a user watching a display unit; an acquisition unit, which is configured to acquire an image captured and acquired through a predetermined camera control; a display control unit, which is configured to display the image acquired by the acquisition unit on the display unit; a first operation unit, which is configured to receive a first operation input from the user; a second operation unit, which is configured to receive a second operation input from the user that is different from the first operation input; and a control unit, which is configured to: in a state where the image is displayed on the display unit through the display control unit, in case of performing the first operation input, perform setting of the predetermined camera control based on the line of sight position of the input line of sight, and in case of performing the second operation input, perform image processing on the image displayed on the display unit based on the line of sight position.

[0009] An electronic device comprises: a line of sight input unit configured to receive a line of sight input performed by the line of sight of a user viewing a display unit; an acquisition unit configured to acquire an image captured and acquired through a predetermined camera control; a display control unit configured to display the image acquired by the acquisition unit on the display unit; a first operation unit configured to receive a first operation input from the user; a second operation unit configured to receive a second operation input from the user that is different from the first operation input; and a control unit configured to: in a state where the image is displayed on the display unit through the display control unit, in case of performing the first operation input, perform setting of the predetermined camera control based on the line of sight position of the input line of sight, and in case of performing the second operation input, perform partial control of the display brightness of the display unit based on the line of sight position.

[0010] A control method for an electronic device, comprising: receiving a line of sight input through the line of sight of a user viewing a display unit; acquiring an image captured and acquired through a predetermined camera control; displaying the acquired image on the display unit; receiving a first operation input from the user; receiving a second operation input from the user that is different from the first operation input; in a state where the image is displayed on the display unit, when the first operation input is performed, setting the predetermined camera control based on the line of sight position of the input line of sight, and in a state where the image is displayed on the display unit, when the second operation input is performed, performing image processing on the image displayed on the display unit based on the line of sight position.

[0011] A control method for an electronic device, comprising: receiving a line of sight input through the line of sight of a user viewing a display unit; acquiring an image captured and acquired through a predetermined camera control; displaying the acquired image on the display unit; receiving a first operation input from the user; receiving a second operation input from the user that is different from the first operation input; in a state where the image is displayed on the display unit, when the first operation input is performed, setting the predetermined camera control based on the line of sight position of the input line of sight, and in a state where the image is displayed on the display unit, when the second operation input is performed, partially controlling the display brightness of the display unit based on the line of sight position.

[0012] A computer-readable medium stores a program for causing a computer to function as each unit of the above-mentioned electronic device.

[0013] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A and Figure 1B This is a diagram of the appearance of a digital camera;

[0015] Figure 2 is a block diagram of a digital camera;

[0016] Figure 3 is a portion of a flowchart depicting image capture mode processing;

[0017] Figure 4 is a portion of a flowchart depicting image capture mode processing;

[0018] Figure 5 is a portion of a flowchart depicting image capture mode processing;

[0019] Figure 6 is an example of changing camera setting values ​​based on the eye-line position; and

[0020] Figure 7 This is an example of AF control based on the eye-line position. DETAILED DESCRIPTION

[0021] External view of digital camera 100

[0022] Preferred embodiments of the present invention will be described with reference to the accompanying drawings. Figure 1A and Figure 1B 1 is an external view of a digital camera 100 , which is an example of an apparatus to which the present invention can be applied. Figure 1A is a front perspective view of the digital camera 100, and Figure 1BIt is a rear perspective view of the digital camera 100 .

[0023] The display unit 28 is located on the back of the digital camera 100 and displays images and various information. The touch panel 70a can detect touch operations performed on the display surface (touch operation surface) of the display unit 28. The viewfinder external display unit 43 is located on the top surface of the digital camera 100 and displays various setting values ​​of the digital camera 100, such as the shutter speed and aperture. The shutter button 61 is an operating member for instructing the capture of an image. The mode selection switch 60 is an operating member for switching between various modes. The terminal cover 40 is a cover for protecting a connector (not shown) that connects the digital camera 100 to an external device.

[0024] The main electronic dial 71 is a rotary operating member, and by rotating it, setting values ​​such as shutter speed and aperture can be changed. The power switch 72 is an operating member for switching the power of the digital camera 100 on and off. The sub-electronic dial 73 is a rotary operating member, and by rotating it, for example, a selection frame (cursor) can be moved and images can be switched. The four-way key 74 is configured so that the upper, lower, left, and right portions of the key can be pressed, respectively, so that processing corresponding to the pressed portion of the four-way key 74 can be performed. The set (SET) button 75 is a push button, and is mainly used to confirm the selected item.

[0025] The video button 76 is used to start or stop capturing (recording) moving images. The AE lock button 77 is a push button, and pressing it in the image capture standby state allows you to fix the exposure. The zoom button 78 is an operation button for switching the zoom mode on or off in the live view display (LV display) in image capture mode. Operating the main electronic dial 71 after setting the zoom mode to "on" allows you to zoom in or out on the live view image (LV image). In playback mode, the zoom button 78 is used as an operation button to zoom in on the reproduced image or increase its magnification. The playback button 79 is an operation button for switching between image capture mode and playback mode. Pressing the playback button 79 in image capture mode changes the mode to playback mode, in which the latest image recorded on the recording medium 200 (described later) is displayed on the display unit 28. The menu button 81 is a push button, and pressing it displays a menu screen that allows you to set various settings on the display unit 28. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four-way key 74, the setting button 75, and the multi-controller (MC) 65. The MC 65 can be operated to indicate one of eight directions and can receive an operation by pressing its center portion.

[0026] The communication terminal 10 is a communication terminal used by the digital camera 100 to communicate with the detachable lens unit 150 (described later). The eyepiece 16 is the eyepiece portion of an eyepiece viewfinder (peep-type viewfinder), and the user can view images displayed on the internal electronic viewfinder (EVF) 29 (described later) through the eyepiece 16. The eye contact detection unit 57 is an eye contact detection sensor for detecting whether the user's eye is in contact with the eyepiece 16. The cover 202 is a cover for the slot for storing the recording medium 200 (described later). The grip 90 is a holding unit with a shape that allows the user to easily hold the digital camera 100 with their right hand to capture images. When the user holds the digital camera 100 by gripping the grip 90 with their pinky, ring, and middle fingers, the mode selection switch 60, shutter button 61, main electronic dial 71, video button 76, and the like are positioned so that they can be operated by the index finger of their right hand. In this state, the MC 65, touch panel 70a, sub-electronic dial 73, four-directional key 74, set button 75, AE lock button 77, zoom button 78, and reproduction button 79 are arranged at positions operable by the right thumb.

[0027] Block diagram of digital camera 100

[0028] Figure 21 is a block diagram illustrating an example of a configuration of a digital camera 100. The lens unit 150 is a lens unit equipped with an interchangeable image capturing lens. The lens 103 is generally composed of a plurality of lenses, but for simplicity, Figure 2 1 is illustrated as a single lens. Communication terminal 6 is a communication terminal for the lens unit 150 to communicate with the digital camera 100, and communication terminal 10 is a communication terminal for the digital camera 100 to communicate with the lens unit 150. The lens unit 150 communicates with the system control unit 50 via these communication terminals 6 and 10. The lens unit 150 then controls the aperture 1 via the aperture drive circuit 2 using the internal lens system control circuit 4. The lens unit 150 also performs focusing by moving the lens 103 via the AF drive circuit 3 using the lens system control circuit 4.

[0029] The shutter 101 is a focal plane shutter that can freely control the exposure time of the image pickup unit 22 based on the control of the system control unit 50 .

[0030] The imaging unit 22 is an image pickup element (image sensor) composed of a CCD or CMOS element, etc., for converting an optical image into an electrical signal. The imaging unit 22 may include an imaging plane phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 23 is used to convert the analog signal output from the imaging unit 22 into a digital signal.

[0031] The image processing unit 24 performs predetermined processing (such as pixel interpolation, resizing (e.g., reduction), or color conversion) on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. Thus, through-the-lens (TTL) type autofocus (AF) processing, automatic exposure (AE) processing, and pre-flash (EF) processing are performed. Furthermore, the image processing unit 24 performs predetermined arithmetic processing using the captured image data, and performs TTL type automatic white balance (AWB) processing based on the obtained arithmetic results.

[0032] The memory control unit 15 controls data transmission / reception between the A / D converter 23, the image processing unit 24, and the memory 32. Output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15. In some cases, the output data from the A / D converter 23 is written to the memory 32 via the memory control unit 15 without using the image processing unit 24. The memory 32 stores image data acquired by the imaging unit 22 and converted into digital data by the A / D converter 23, and stores image data to be displayed on the display unit 28 or the EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined duration of moving images and sound.

[0033] The memory 32 is also a memory (video memory) for image display. Image data for display written to the memory 32 is displayed on the display unit 28 or the EVF 29 via the memory control unit 15. The display unit 28 and the EVF 29 each display on a display such as an LCD or an organic EL based on a signal from the memory control unit 15. If the data A / D-converted by the A / D converter 23 and stored in the memory 32 is sequentially transmitted and displayed on the display unit 28 or the EVF 29, a live view display (LV display) can be performed. Hereinafter, the image displayed on the live view display is referred to as a live view image (LV image).

[0034] The visual line detection unit 160 detects the user's visual line at the eyepiece 16. The visual line detection unit 160 is composed of a dichroic mirror 162, an imaging lens 163, a visual line detection sensor 164, a visual line detection circuit 165, and an infrared light emitting diode 166. Since the system control unit 50 can perform predetermined processing based on the detection of the visual line, the visual line detection unit 160 can be regarded as a part of the operation unit 70.

[0035] In this embodiment, the line of sight detection unit 160 detects the line of sight using the corneal reflection method. The corneal reflection method is a method for detecting the direction and position of the line of sight based on the positional relationship between the reflected light (particularly light reflected by the cornea) generated when the infrared light emitted from the infrared light emitting diode 166 is reflected by the eyeball (eye) 161 and the pupil of the eyeball (eye) 161. As for the method used by the line of sight detection unit 160, not only the corneal reflection method can be used, but also various other methods for detecting the direction and position of the line of sight can be used. For example, the sclera reflection method can be used, which uses the difference in light reflectivity between the iris and the white of the eye.

[0036] The infrared light emitting diode 166 is a light emitting element for detecting the user's line of sight position in the viewfinder screen, and emits infrared light toward the eyeball (eye) 161 of the user in contact with the eyepiece 16. The infrared light emitted from the infrared light emitting diode 166 is reflected by the eyeball (eye) 161, and the infrared reflected light reaches the dichroic mirror 162. The dichroic mirror 162 reflects only infrared light and allows visible light to pass through. The infrared reflected light whose optical path has been changed forms an image on the imaging surface of the line of sight detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical component that constitutes the line of sight detection optical system. The line of sight detection sensor 164 is composed of a camera device such as a CCD-type image sensor.

[0037] The line of sight detection sensor 164 performs photoelectric conversion on the input infrared reflected light and outputs the generated electrical signal to the line of sight detection circuit 165. The line of sight detection circuit 165 includes at least one processor and detects the user's line of sight position based on the image or movement of the user's eyeball (eye) 161 based on the output signal of the line of sight detection sensor 164, and outputs the detected information to the system control unit 50.

[0038] Various setting values ​​of the camera, such as shutter speed and aperture, are displayed on the viewfinder external display unit 43 via the viewfinder external display unit drive circuit 44 .

[0039] The nonvolatile memory 56 is an electrically erasable and recordable memory such as a Flash ROM. In the nonvolatile memory 56, for example, constants and programs for operating the system control unit 50 are recorded. Here, "program" refers to a computer program for executing various flowcharts described later in this embodiment.

[0040] The system control unit 50 is a control unit composed of at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 implements the various processing steps of this embodiment (described later) by executing programs stored in the nonvolatile memory 56. The system memory 52 is, for example, a RAM, and the system control unit 50 stores constants and variables used to operate the system control unit 50 and programs read from the nonvolatile memory 56 in the system memory 52. ​​The system control unit 50 also controls the display by controlling the memory 32, the display unit 28, the EVF 29, and the like.

[0041] The system timer 53 is a clock unit that measures time for various controls and time of an internal clock.

[0042] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, and a switch circuit for switching the blocks to be energized. For example, it detects whether a battery is installed, the type of battery, and the remaining battery charge. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage to each unit including the recording medium 200 for the required period of time. The power supply unit 30 is composed of a primary battery (e.g., an alkaline battery, a lithium battery), a secondary battery (e.g., a NiCd battery, a NiMH battery, a Li battery), an AC adapter, and the like.

[0043] The recording medium I / F 18 is an interface with a recording medium 200 (eg, a memory card, a hard disk). The recording medium 200 is a recording medium (such as a memory card) for recording captured images and is composed of a semiconductor memory, a magnetic disk, or the like.

[0044] The communication unit 54 transmits and receives video signals and sound signals to and from an external device connected wirelessly or via a cable. The communication unit 54 can also be connected to a wireless local area network (LAN) and the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth low energy. The communication unit 54 can transmit images (including LV images) captured by the camera unit 22 and images recorded on the recording medium 200, and can receive image data and various other information from external devices.

[0045] The posture detection unit 55 detects the posture of the digital camera 100 relative to the direction of gravity. Based on the posture detected by the posture detection unit 55, it is possible to determine whether the image captured by the imaging unit 22 was captured by the digital camera 100 held horizontally or held vertically. The system control unit 50 can append posture information based on the posture detected by the posture detection unit 55 to the image file of the image captured by the imaging unit 22, or can rotate and record the image. The posture detection unit 55 can use an acceleration sensor, a gyro sensor, or the like. The acceleration sensor or gyro sensor of the posture detection unit 55 can be used to detect the movement of the digital camera 100 (e.g., panning, tilting, raising, or stillness).

[0046] The eye contact detection unit 57 is an eye contact detection sensor for detecting the eye (object) 161 approaching (contacting) or withdrawing (releasing) from the eyepiece 16 of the eyepiece finder (viewfinder). The system control unit 50 switches the display unit 28 and EVF 29 between display (display state) and non-display (non-display state) based on the state detected by the eye contact detection unit 57. Specifically, when the digital camera 100 is at least in the image capture standby state and the display destination switching setting for the live view image captured by the imaging unit 22 is set to automatic switching, in the non-eye contact state, the display destination is set to the display unit 28 (display of the display unit 28 is turned on), and the EVF 29 is set to non-display. On the other hand, in the eye contact state, the display destination is set to the EVF 29 (display of the EVF 29 is turned on), and the display unit 28 is set to non-display. For example, an infrared proximity sensor can be used for the eye contact detection unit 57 to detect the proximity of an object to the eyepiece 16 of the viewfinder including the EVF 29. When an object approaches, infrared light emitted from the light emitting unit (not shown) of the eye contact detection unit 57 is reflected by the object and received by the light receiving unit (not shown) of the infrared proximity sensor. The distance from the object to the eyepiece 16 (eyepiece distance) can also be determined by the amount of received infrared light. In this way, the eye contact detection unit 57 performs eye contact detection to detect the proximity distance of the object to the eyepiece 16. In this embodiment, the light emitting unit and the light receiving unit of the eye contact detection unit 57 are assumed to be separate devices from the infrared light emitting diode 166 and the line of sight detection sensor 164 described above. However, the function of the light emitting unit of the eye contact detection unit 57 can be provided by the infrared light emitting diode 166, and the light receiving unit of the eye contact detection unit 57 can be provided by the line of sight detection sensor 164. In addition, in this embodiment, it is assumed that eye contact is detected when an object is detected approaching the eyepiece 16 less than a predetermined distance after the non-eye contact state (non-proximity state). It is also assumed that eye withdrawal is detected when the object whose approach is detected moves away from the eye contact state (proximity state) by at least a predetermined distance. For example, the threshold for detecting eye contact and the threshold for detecting eye withdrawal can be distinguished by hysteresis. After eye contact is detected, it is assumed that the eye contact state continues until eye withdrawal is detected. Then, after eye withdrawal is detected, it is assumed that the non-eye contact state continues until eye contact is detected. It should be noted that the infrared proximity sensor is merely an example, and other types of sensors can be used for the eye contact detection unit 57 as long as the sensor can detect the approach of an eye or object that can be regarded as eye contact.

[0047] The system control unit 50 can detect the following operations performed on the eyepiece 16 or the states thereof based on the output from the visual line detection unit 160 .

[0048] The line of sight that has not been directed toward the eyepiece 16 is directed toward the eyepiece 16, i.e., line of sight input begins.

[0049] The line of sight is being input into the eyepiece 16 .

[0050] The eyepiece 16 is being looked at (for example, a state where the user's line of sight position does not move more than a predetermined amount of movement within a predetermined time).

[0051] The sight line that is being directed toward the eyepiece 16 is turned away, that is, the sight line input is terminated.

[0052] The line of sight is not input into the eyepiece 16.

[0053] These operations, states, and the position (direction) of the line of sight relative to the eyepiece 16 are notified to the system control unit 50 via the internal bus, and based on the notified information, the system control unit 50 determines the type of operation (line of sight operation) performed on the eyepiece 16.

[0054] The operation unit 70 is an input unit that receives an operation performed by a user (user operation) and is used to input various operation instructions to the system control unit 50. Figure 2 As illustrated in FIG. 1 , the operation unit 70 includes a mode selection switch 60, a shutter button 61, a power switch 72, a touch panel 70a, and other operating members 70b. The other operating members 70b include an MC 65, a main electronic dial 71, a sub-electronic dial 73, a four-directional key 74, a set button 75, a video button 76, an AE lock button 77, a zoom button 78, a reproduction button 79, and a menu button 81.

[0055] The mode selection switch 60 switches the operating mode of the system control unit 50 to a still image shooting mode, a moving image shooting mode, or a reproduction mode, etc. The still image shooting mode includes the following modes: automatic image shooting mode, automatic scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). Various scene modes and custom modes for making image shooting settings for various image shooting scenes are also included. The user can directly select any of these modes using the mode selection switch 60. The user can also first use the mode selection switch 60 to select the image shooting mode list screen, and then use another operating member to select any of the multiple modes displayed on the list. In the same way, the moving image shooting mode can include multiple modes.

[0056] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. During an intermediate operation of the shutter button 61, that is, in the half-pressed state (image capture preparation instruction), the first shutter switch 62 is turned on and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 initiates image capture preparation operations such as autofocus (AF), auto exposure (AE), auto white balance (AWB), and pre-flash (EF). When the operation of the shutter button 61 is completed, that is, in the fully pressed state (image capture instruction), the second shutter switch 64 is turned on and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 initiates a series of image capture operations, from reading a signal from the imaging unit 22 to writing the captured image as an image file to the recording medium 200.

[0057] The touch panel 70a and the display unit 28 can be integrated. For example, the touch panel 70a is configured so that light transmittance does not interfere with the display on the display unit 28, and is superimposed on the upper layer of the display surface of the display unit 28. Then, the input coordinates of the touch panel 70a are made to correspond to the display coordinates on the display surface of the display unit 28. In this way, a graphical user interface (GUI) that allows the user to directly operate the screen displayed on the display unit 28 can be provided. The system control unit 50 can detect the following operations on the touch panel 70a or its status.

[0058] A finger or pen that was not touching the touch panel 70 a touches the touch panel 70 a , that is, touch-up starts (hereinafter referred to as “touch-down”).

[0059] A finger or a pen is touching the touch panel 70a (hereinafter referred to as "Touch-On").

[0060] A finger or a pen moves while touching the touch panel 70 a (hereinafter referred to as “Touch-Move”).

[0061] The finger or pen touching the touch panel 70 a is released from the touch panel 70 a , that is, the touch ends (hereinafter referred to as “Touch-Up”).

[0062] Nothing is touching the touch panel 70a (hereinafter referred to as "Touch-Off").

[0063] When "touch" is detected, "touch persist" is also detected. "Touch persist" is always detected unless "touch stop" is detected after "touch." Similarly, when "touch move" is detected, "touch persist" is also detected. Even if "touch persist" is detected, "touch move" is not detected unless the touch position is moving. "Not touch" is detected when "touch stop" is detected for all fingers or pens.

[0064] These operations, states, and the coordinates of the position of the finger or pen touching the touch panel 70a are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 then determines the type of operation (touch operation) performed on the touch panel 70a. Regarding "touch movement," the system control unit 50 can also determine the direction of movement of the finger or pen moving on the touch panel 70a based on changes in the position coordinates, both for the vertical component and the horizontal component on the touch panel 70a. If a "touch movement" of at least a predetermined distance is detected, the system control unit 50 determines that a slide operation has been performed. The operation of quickly moving a finger a certain distance on the touch panel 70a while in a touch state and then releasing it is called a "flick." In other words, a flick is an operation of quickly moving a finger on the touch panel 70a and then releasing it. If a "touch movement" of at least a predetermined distance at at least a predetermined speed is detected and a "touch stop" is subsequently detected, the system control unit 50 determines that a flick has been performed (determined to have been performed after a slide operation). In addition, a touch operation of touching multiple points (for example, 2 points) at the same time and moving these touch positions closer together is called "pinch-in", and a touch operation of moving these touch positions farther apart is called "pinch-out". "Pinch-in" and "pinch-out" are collectively referred to as a pinch operation (or simply "pinch-out"). For the touch panel 70a, various types of touch panels can be used, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. There is a method of detecting a touch when the touch panel is actually contacted and a method of detecting a touch when a finger or a pen approaches the touch panel, and either method can be used.

[0065] Based on user operations on the operation unit 70, the system control unit 50 can set the method of specifying an index position (e.g., an AF position (the position of the AF frame)) using a touch operation to absolute position specification or relative position specification. For example, in the case of absolute position specification, when the touch panel 70a is touched, the system control unit 50 sets an index position corresponding to the touched position. In other words, the coordinates of the position where the touch operation was performed (input coordinates) correspond to the coordinates of the position on the display unit 28 (display coordinates). On the other hand, in the case of relative position specification, the coordinates of the position where the touch operation was performed do not correspond to the coordinates of the position on the display unit 28. In relative position specification, regardless of the position of "touch" on the touch panel 70a, the system control unit 50 moves the index position from the currently set index position in the movement direction of "touch move" by a distance corresponding to the movement amount of "touch move".

[0066] Operation of the digital camera 100

[0067] Figure 3 and Figure 4 Indicates a flowchart for describing details related to image capture mode processing performed by the digital camera 100. This processing is implemented by the system control unit 50 expanding a program recorded in the non-volatile memory 56 in the system memory 52 and executing the program. For example, when the digital camera 100 starts in image capture mode, or when the system control unit 50 switches the current mode to image capture mode, the system control unit 50 initializes flags and control variables, etc., and then displays a live view image captured by the imaging unit 22 on the display unit 28 and the EVF 29. In addition, based on the camera setting value, the system control unit 50 displays an information icon indicating the camera setting value as superimposed on the live view image. Then, the system control unit 50 starts Figure 3 and Figure 4 Image capture mode processing in .

[0068] exist Figure 3 In S301 in FIG. 5 , the system control unit 50 determines whether an operation for switching enable / disable of the line of sight detection function has been performed on the operation unit 70. If so, the process proceeds to S302, or if not, to S303.

[0069] In S302 , the system control unit 50 changes the camera setting value according to the operation in S301 so that enable / disable of the visual line detection function is switched.

[0070] In S303, the system control unit 50 determines whether an operation has been performed on the operation unit 70 to switch the display of the line of sight pointer between on and off. If so, the process proceeds to S304; if not, the process proceeds to S305. The line of sight pointer is a display item (indicator) that indicates the line of sight position (the position where the line of sight is input) in the eyepiece 16 (display surface of the EVF 29) and is displayed at the line of sight position.

[0071] In S304 , the system control unit 50 changes the camera setting value according to the operation in S303 , so that enable / disable of the line-of-sight pointer display is switched.

[0072] In S305, the system control unit 50 determines whether an operation has been performed on the operation unit 70 to switch between enabling and disabling the line-of-sight position determination function for determining the line-of-sight position based on the pressing (turning on) of the first shutter switch 62. If such an operation has been performed, the process proceeds to S306, and if such an operation has not been performed, the process proceeds to S307.

[0073] In S306 , the system control unit 50 changes the camera setting value according to the operation in S305 , so that enable / disable of the line-of-sight position determination function is switched.

[0074] In S307, the system control unit 50 determines whether an operation for switching the setting of the AF mode has been performed on the operation unit 70. If so, the process proceeds to S308, or if not, to S309.

[0075] In S308 , the system control unit 50 changes the camera setting value according to the operation in S307 , so that the setting of the AF mode is switched.

[0076] In S309, the system control unit 50 determines whether the sight line detection function is enabled. If enabled, the process proceeds to S310, or if not enabled, proceeds to S312 ( Figure 4 ).

[0077] In S310, the system control unit 50 determines whether the sight pointer display is enabled. If enabled, the process proceeds to S311, or if not enabled, proceeds to S312 ( Figure 4 ).

[0078] In S311 , the system control unit 50 acquires information about the line-of-sight position from the line-of-sight detection unit 160 , and displays the line-of-sight pointer at the line-of-sight position on the display surface of the EVF 29 .

[0079] In S312, the system control unit 50 acquires the current lens ID (identification information (individual information) for identifying the lens unit 150) and the currently set aperture value. For example, the system control unit 50 acquires the lens ID and aperture value from the lens unit 150. The system control unit 50 may have previously acquired the lens ID and aperture value from the lens unit 150 and recorded them in the system memory 52. ​​Then, in S312, the system control unit 50 may acquire the lens ID and aperture value from the system memory 52.

[0080] In S313, the system control unit 50 determines whether the AF target fixation setting is enabled. The AF target fixation setting is a setting (function) for fixing the AF target (the subject to be focused on; the subject for which the focus (focus point) is set). If enabled, the process proceeds to S314, and if not, to S315. The AF target can be regarded as a focus detection area used to focus on the corresponding subject.

[0081] The system control unit 50 can switch the AF target fixation setting between enabled and disabled based on an operation (operation input) performed on the operation unit 70. For example, the AF target fixation setting can be switched between enabled and disabled based on an operation of an operating member (such as the AF-ON button, AEL button, AF-SEL button, touch panel, built-in controller, OTP, and preview button) that can be operated while the shutter button 61 is pressed. The operating member for enabling the AF target fixation setting and the operating member for disabling the AF target fixation setting may or may not be different from each other. The system control unit 50 can switch the AF target fixation setting between enabled and disabled by rotating the operating member based on an operation performed on a specific operating member (one operating member). Specifically, if the AF target fixation setting is disabled, the system control unit 50 can enable the AF target fixation setting based on an operation of the specific operating member, and if the AF target fixation setting is enabled, the system control unit 50 can disable the AF target fixation setting based on an operation of the specific operating member. The user can switch the AF target fixation setting between enabled and disabled on the menu screen.

[0082] In this embodiment, when the AF target fixation setting is disabled, the system control unit 50 determines the subject displayed at the line of sight position (the position at which the line of sight was input) in the live view image as the AF target and controls the focus to be on the determined subject. Therefore, if the line of sight position changes to view a subject different from the AF target subject, the AF target is switched to the subject located at the line of sight position. Then, when the AF target fixation setting is switched from disabled to enabled, the system control unit 50 fixes the AF target to the focused subject (the AF target before the AF target fixation setting was switched) and controls the focus to continue on that subject. Similarly, when the AF target fixation setting is enabled, even if the line of sight position changes to view a subject different from the AF target subject, the AF target is not switched.

[0083] In S314, the system control unit 50 fixes the AF target regardless of the line of sight position. Thus, even if the line of sight deviates from the AF target, the AF target does not change, so the framing (composition) can be checked and the user's intended operation can be performed.

[0084] In S315, the system control unit 50 determines the AF target according to the line of sight position. If the line of sight position changes to view a subject different from the AF target subject, the system control unit 50 switches the AF target to the subject at the line of sight position.

[0085] Depending on the lens unit 150 (e.g., lens type, aperture), the brightness of the live view image may decrease from the center to the edge of the live view image (peripheral darkening). In S316, the system control unit 50 calculates the amount of brightness reduction due to peripheral darkening (the amount of brightness reduction of the image from the center to the edge of the image) based on the lens ID and aperture value acquired in S312, and determines whether the amount of brightness reduction is at least a predetermined threshold. The amount of brightness reduction may be calculated in S312. If the amount of brightness reduction is at least the threshold, the process proceeds to S317. If the amount of brightness reduction is not at least the threshold (if the amount of brightness reduction is less than the threshold), the process proceeds to S313.

[0086] In S317, the system control unit 50 uses the image processing unit 24 to detect the region of the subject located at the line of sight in the live view image. For example, the system control unit 50 divides the live view image into a predetermined number of regions and calculates the similarity of each region to the region at the line of sight based on information related to the color and brightness of each region. The system control unit 50 then determines that the region with a similarity greater than a predetermined threshold is the region of the subject located at the line of sight.

[0087] In S318, the system control unit 50 uses the image processing unit 24 to generate a mask to distinguish the area detected in S317 (the sight line area) from other areas. The mask is used to process only the sight line area and disable processing for other areas (areas different from the sight line area). In this embodiment, a bitmap image is generated as the mask, in which the pixel values ​​of the sight line area are 1 and the pixel values ​​of other areas are 0.

[0088] The sight area is not limited to the area of ​​the subject located at the sight position. For example, the sight area may be an area that includes the subject located at the sight position and is larger than the subject. The sight area may be other areas that include the sight position, such as an area that is not more than a predetermined distance (a distance corresponding to a predetermined number of pixels) from the sight position and is composed of a plurality of pixels. The size of the sight area may be a predetermined fixed size, or may be a size that is determined or changed according to the display size (size on the live view image) of the subject located at the sight position. For example, as the display size of the subject located at the sight position becomes larger, the sight area may become larger. The shape of the sight area is not particularly limited, and may be circular (e.g., a perfect circle, an ellipse), or may be polygonal (e.g., a triangle, a hexagon).

[0089] In S319, the system control unit 50 uses the image processing unit 24 to change the brightness of the line of sight area (the area based on the line of sight position) specified by the mask generated in S318 in the live view image. In the present embodiment, the system control unit 50 changes (increases) the brightness of the line of sight area by the amount of change in the brightness reduction amount (calculated in S316; the amount of light based on the image height) based on the peripheral darkening. The brightness of the line of sight area (pixel) is changed by multiplying the pixel value by a gain value, adding an offset value to the pixel value, etc. Thus, for example, when the brightness of the line of sight area is lower than a threshold value, the brightness of the line of sight area is corrected to the brightness of the threshold value. The brightness of other areas (areas different from the line of sight area) is not changed. There is no particular limitation on the method of changing the brightness of the line of sight area. For example, the brightness of each pixel in the line of sight area may be increased or decreased uniformly, or may be set to be closer to a predetermined brightness.

[0090] In this embodiment, the brightness of the sight line area (the area based on the sight line position) is controlled by image processing of the displayed image, but it is also possible to control it by local brightness control of the display device body (partial control of the display brightness). For example, the display unit 28 and EVF 29 can each locally control the display brightness based on the size signal of the sight line area on a display such as an LCD or organic EL.

[0091] Furthermore, in this embodiment, the brightness of the image in the line of sight area is controlled. However, if the image in the line of sight area can be partially processed, the purpose of the present invention can also be achieved by a different method. Examples of image processing that is partially performed on the image in the line of sight area are: white balance, color conversion, magnification display processing; and sharpening processing. By changing the image processing parameters in the line of sight area, the image processing itself can be performed on the entire image. The system control unit 50 uses the image processing unit 24 to partially perform image processing on the line of sight area (the area based on the line of sight position) specified by the mask generated in S318 in the live view image, and sequentially displays the processed images as live view images on the EVF 29.

[0092] At the beginning Figure 3 and Figure 4 After the image capture mode processing in

[15] is completed, the system control unit 50 reads the signal from the imaging unit 22 via the second shutter switch signal SW2 and writes the captured image as an image file to the recording medium 200. In this case, the brightness control (partial image processing) for the visual field area (pixels) described above is not performed on the image to be recorded. However, both the image that has undergone partial image processing and the image that has not undergone partial image processing can be recorded as a recorded image.

[0093] As described above, according to this embodiment, focus control using line-of-sight input (control to focus on the subject based on the line-of-sight position) is performed when the AF target fixation setting is disabled, but not when the AF target fixation setting is enabled. On the other hand, brightness control using line-of-sight input (control of the brightness of the line-of-sight area) is performed regardless of whether the AF target fixation setting is enabled. Therefore, focus control using line-of-sight input and brightness control can be performed appropriately (with good operability). For example, when only brightness control using line-of-sight input is performed, focus control using line-of-sight input can be prevented from being performed contrary to the user's intention.

[0094] Although focus control and brightness control using line-of-sight input have been described, the settings of imaging parameters can be changed by selecting information icons displayed on a live view image using line-of-sight input and operating electronic dials and buttons. Figure 6 As illustrated in FIG, when the user is gazing at an information icon (e.g., shutter speed, aperture) indicating a camera setting value displayed on a live view image, the system control unit 50 displays the gaze information icon in a selected state (e.g., highlighted) on the EVF 29. In this state, for example, the user can change the setting value of the imaging parameter (e.g., shutter speed, aperture, focal length) corresponding to the information icon by rotating the electronic dial.

[0095] In the above description, the above-described various controls are performed by the system control unit 50 , but may be performed by one hardware component or by a plurality of hardware components (eg, a plurality of processors and circuits) that share processing to control the entire apparatus.

[0096] Although the present invention has been described with reference to preferred embodiments, the present invention is not limited to these specific embodiments and includes various modes within the scope not departing from the essence of the present invention. In addition, the above-mentioned various embodiments are only examples of the present invention and can be combined as needed.

[0097] For example, the case where the brightness of the sight line area is changed when the condition that the brightness reduction amount of the peripheral darkening is at least a predetermined threshold is satisfied is described, but the brightness of the sight line area may be changed regardless of whether this condition is satisfied. Figure 4 In the example described above, steps S317 to S319 can be always processed without S316. Other conditions can be used to determine whether the brightness of the sight line area has changed. Furthermore, although an example has been described in which the AF target subject is fixed without determining the AF target based on the sight line position, the AF position (the position to be focused on) can be fixed without determining the AF target based on the sight line position. Furthermore, although an example has been described in which the enabling / disabling of various settings (e.g., the AF target fixing setting) is switched based on user operations, the enabling / disabling of various settings can be switched automatically based on the image capture scene, etc.

[0098] As another AF target control method, instead of always using the line of sight position for AF target setting as in the case of this embodiment, the AF target setting may be switched by a user operation. For example, when the shutter switch signal SW1 (or an operation from another independent operating member) is input, as in Figure 7 As illustrated in , the AF frame can be moved to a subject displayed in a line-of-sight position so that the subject can be determined as an AF target.

[0099] replace Figure 4 The processing in Figure 5 According to the processing in Figure 5 In the processing, focus control using line of sight input or brightness control using line of sight input is selectively performed, thereby improving operability and user-friendliness.

[0100] In S320, the system control unit 50 determines whether the AF target fixing setting is enabled. If enabled, the process proceeds to S321, or if not enabled, proceeds to S322.

[0101] In S321 , the system control unit 50 fixes the AF target regardless of the line-of-sight position.

[0102] In S322 , the system control unit 50 determines an AF target according to the line-of-sight position.

[0103] In S323, the system control unit 50 determines whether the brightness correction setting is enabled. The brightness correction setting is a setting for changing (correcting) the brightness of the visual field. If enabled, the process proceeds to S324, and if not, to S326. As in the case of the AF target fixing setting, the system control unit 50 switches the brightness correction setting on / off based on the operation of the operation unit 70. It should be noted that the operating member for switching the brightness correction setting on / off is different from the operating member for switching the AF target fixing setting on / off.

[0104] In S324, the system control unit 50 generates a mask to distinguish the sight line area from other areas using the image processing unit 24. For example, an area that is not more than a predetermined distance (a distance corresponding to a predetermined number of pixels) from the sight line position and is composed of a plurality of pixels is regarded as the sight line area, and a bitmap image is generated as the mask in which the pixel value of the sight line area is 1 and the pixel values ​​of other areas are 0.

[0105] In step S325, the system control unit 50 uses the image processing unit 24 to change the brightness of the visual field area specified by the mask generated in step S324 in the live view image. For example, the system control unit 50 increases the brightness of areas with brightness lower than a predetermined brightness to the predetermined brightness. The brightness of other areas (areas other than the visual field area) remains unchanged.

[0106] In S326, the system control unit 50 cancels the brightness correction of the visual line area using the image processing unit 24. Thus, the brightness of the visual line area becomes the brightness before the change in S325.

[0107] In the above embodiment, the case where the present invention is applied to a digital camera (camera) is described as an example, but the present invention is not limited to this, but is applicable to any electronic device that can receive line of sight input. For example, the present invention is applicable to personal computers, PDAs, portable telephone terminals, portable image viewers, printers, digital photo frames, music players, game consoles, e-book readers, and video players. The present invention is also applicable to wearable devices (for example, head-mounted displays (HMDs)), display devices (including projectors), tablet terminals, smart phones, AI speakers, home appliances, vehicle-mounted devices, and medical equipment. The processing for receiving line of sight input may be a process of detecting a line of sight, or may be a process of acquiring information related to a line of sight detected by an external device. It is only required that the electronic device to which the present invention is applied acquires an image captured by a predetermined camera control, and the image capture itself can be performed by an external device.

[0108] According to the present disclosure, a plurality of controls (processing steps) such as focus control and brightness control using line-of-sight input can be performed appropriately (with good operability).

[0109] Other embodiments

[0110] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0111] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An electronic device comprising: a sight line input unit configured to receive a sight line input performed by a user viewing the display unit; an acquisition unit configured to acquire an image photographed and acquired by predetermined imaging control; a display control unit configured to display the image acquired by the acquisition unit on the display unit; a first operating unit configured to receive a first operating input from the user that is different from the sight input; a second operation unit configured to receive a second operation input from a user that is different from the first operation input; as well as a control unit configured to: in a state where the image is displayed on the display unit by the display control unit, In the case where the first operation input is performed, the predetermined imaging control setting is performed based on the line of sight position input, wherein the predetermined imaging control setting is a setting of a focus detection area for focusing on a subject corresponding to the line of sight position, and In the case where the second operation input is performed, the peripheral darkening brightness reduction amount as the brightness reduction amount of the image from the center to the edge of the image is calculated based on the aperture value set for the image capturing lens used to capture the image, and image processing is performed on the image displayed on the display unit based on the line of sight position, wherein the image processing is processing for increasing the brightness of the area corresponding to the line of sight position in the image displayed on the display unit by an amount of change based on the peripheral darkening brightness reduction amount.

2. The electronic device according to claim 1, wherein When the brightness of the area is lower than a threshold value, the control unit corrects the brightness of the area to brightness of the threshold value.

3. The electronic device according to claim 1 or 2, wherein: The image processing is a process for changing an image processing parameter of a region corresponding to the line-of-sight position in the image displayed on the display unit.

4. The electronic device according to claim 1 or 2, wherein: The image processing is performed on a region corresponding to the sight line position in the image displayed on the display unit.

5. The electronic device according to claim 1 or 2, wherein: The control unit In a case where the second operation input is performed, performing image processing on the image displayed on the display unit based on the line of sight position, and The image to be recorded is recorded in the recording medium without performing the image processing.

6. The electronic device according to claim 1 or 2, wherein: The display control unit displays an indicator corresponding to the sight line position.

7. A method for controlling an electronic device, comprising: receiving a line of sight input by a user viewing the display unit; Acquiring images captured and obtained through predetermined camera control; displaying the acquired image on the display unit; receiving a first operation input from the user that is different from the sight input; receiving a second operation input from a user that is different from the first operation input; In a state where the image is displayed on the display unit, when the first operation input is performed, setting of the predetermined imaging control is performed based on the line of sight position of the input line of sight, wherein the predetermined imaging control setting is setting of a focus detection area for focusing on a subject corresponding to the line of sight position, and With the image displayed on the display unit, when the second operation input is performed, the peripheral darkening brightness reduction amount as the brightness reduction amount of the image from the center to the edge of the image is calculated based on the aperture value set for the image capturing lens used to capture the image, and image processing is performed on the image displayed on the display unit based on the line of sight position, wherein the image processing is a process for increasing the brightness of the area corresponding to the line of sight position in the image displayed on the display unit by an amount of change based on the peripheral darkening brightness reduction amount. 8 . A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 7 .

9. A computer program product comprising a program for causing a computer to execute the control method according to claim 7.

Images