Electronic device

Abstract

To provide an electronic apparatus capable of preferably (good operability) performing a plurality of controls (processes) such as focusing control and brightness control by line-of-sight input.SOLUTION: An electronic apparatus according to the present invention includes line-of-sight input means that accepts line-of-sight input by the line of sight of a user who looks at display means, acquisition means that acquires an image acquired by a predetermined imaging control, display control means that displays the image acquired by the acquisition means on the display means, first operation means that accepts a first operation input by the user, second operation means that accepts a second operation input different from the first operation input by the user, and control means that sets predetermined imaging control on the basis of a line-of-sight position in which a line-of-sight input is received when the first operation input is made in a state in which the image is displayed on the display means by the display control means, and performs image control on the image displayed on the display means on the basis of the line-of-sight position when the second operation input is made.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an electronic device, and more particularly to an electronic device capable of receiving eye-gaze input. [Background technology]

[0002] There are electronic devices that users can operate by looking at their eyes (hereinafter referred to as "gaze input"). Gaze input is particularly effective when a user wants to give instantaneous operating instructions to electronic devices such as digital cameras and game consoles.

[0003] Patent document 1 discloses a method for determining the user attention area that the user is paying attention to, and based on the determination result, controlling the brightness of the user attention area in the game image to differentiate between the brightness of the user attention area and the brightness of non-attention areas other than the user attention area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-197109 A Summary of the Invention [Problem to be solved by the invention]

[0005] As a camera function (technology), a function has been proposed that constantly controls focus by eye gaze input (controls the camera to accept eye gaze input and adjusts the focus at the AF frame position according to the eye gaze position (the position where the eye gaze input was made) during continuous shooting, etc. With this function, the user needs to keep looking at the subject on which they want to focus (bring in focus) in the displayed live view image without taking their eyes off the subject.

[0006] Furthermore, by using the technology disclosed in Patent Document 1, it becomes possible to control the brightness of a region of interest in a live view image by gaze input (brightness control by gaze input).

[0007] However, adding a function of brightness control by eye gaze input to an electronic device that always performs focus control by eye gaze input may result in poor focus control (good operability). For example, assume that a user looks at subject B in a live view image in order to control the brightness of subject B, which is different from subject A on which the user wishes to focus. In this case, the brightness of subject B can be controlled as the user intends, but the focus target switches from subject A to subject B against the user's intention. In addition, the user may check a subject other than the one on which the user wishes to focus in order to check the framing (composition). When such a check is performed, the focus target also switches against the user's intention.

[0008] An object of the present invention is to provide an electronic device that can suitably (with good operability) perform a plurality of controls (processes) such as focus control and brightness control by eye-gaze input. [Means for solving the problem]

[0009] The electronic device of the present invention includes: a gaze input unit that receives gaze input by the gaze of a user looking at a display unit; an acquisition unit that acquires an image captured by a predetermined imaging control; a display control unit that displays the image acquired by the acquisition unit on the display unit; Hand To A first operation input is received by the Materials and the user Hand a second operation input that is different from the first operation input by Materials The image is displayed on the display means by the display control means. A gaze input is input to a part of the image. When the first operation input is made in the state where the The portion of the image and when the second operation input is received, The above image The above part and a control means for performing image processing on the basis of the detected image. Effect of the Invention

[0010] According to the present invention, a plurality of controls (processes) such as focus control and brightness control based on eye-gaze input can be preferably performed (with good operability). [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an external view of a digital camera. [Diagram 2] FIG. 1 is a block diagram of a digital camera. [Diagram 3] 11 is a part of a flowchart of a shooting mode process. [Figure 4] 11 is a part of a flowchart of a shooting mode process. [Diagram 5] 11 is a part of a flowchart of a shooting mode process. [Figure 6] 13 is an example of changing image capture setting values ​​depending on the gaze position. [Figure 7] This is an example of AF control based on gaze position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <External view of digital camera 100> A preferred embodiment of the present invention will be described below with reference to the drawings. Figures 1(A) and 1(B) show the appearance of a digital camera 100 as an example of a device to which the present invention can be applied. Figure 1(A) is a front perspective view of the digital camera 100, and Figure 1(B) is a rear perspective view of the digital camera 100.

[0013] The display unit 28 is provided on the rear surface of the digital camera 100 and displays images and various information. The touch panel 70a can detect touch operations on the display surface (touch operation surface) of the display unit 28. The outside-finder display unit 43 is provided on the top surface of the digital camera 100 and displays various settings of the digital camera 100, including the shutter speed and aperture. The shutter button 61 is an operating member for issuing shooting instructions. The mode change switch 60 is an operating member for switching between various modes. The terminal cover 40 is a cover for protecting a connector (not shown) that connects the digital camera 100 to an external device.

[0014] The main electronic dial 71 is a rotary operation member, and by turning the main electronic dial 71, settings such as the shutter speed and aperture can be changed. The power switch 72 is an operation member that switches the power of the digital camera 100 between ON and OFF. The sub electronic dial 73 is a rotary operation member, and by turning the sub electronic dial 73, the selection frame (cursor) can be moved and images can be forwarded. The four-way key 74 is configured so that the up, down, left, and right parts can each be pressed, and processing can be performed according to the part of the four-way key 74 that is pressed. The SET button 75 is a push button, and is mainly used to confirm a selection item.

[0015] The video button 76 is used to start and stop video shooting (recording). The AE lock button 77 is a push button, and by pressing the AE lock button 77 in shooting standby mode, the exposure state can be fixed. The enlargement button 78 is an operation button for switching the enlargement mode ON and OFF in the live view display (LV display) in shooting mode. By turning the enlargement mode ON and then operating the main electronic dial 71, the live view image (LV image) can be enlarged or reduced. In playback mode, the enlargement button 78 functions as an operation button for enlarging the playback image and increasing its magnification. The playback button 79 is an operation button for switching between shooting mode and playback mode. Pressing the playback button 79 in shooting mode transitions to playback mode, and the recording medium 20 The most recent image among the images recorded in memory card 50 (described later) can be displayed on display unit 28. Menu button 81 is a push button, and when menu button 81 is pressed, a menu screen on which various settings can be made is displayed on display unit 28. The user can intuitively make various settings using the menu screen displayed on display unit 28, the four-way key 74, the SET button 75, the multi-controller (MC) 65, etc. MC 65 can accept directional input operations in eight directions and pressing in the center.

[0016] The communication terminal 10 is a communication terminal for the digital camera 100 to communicate with the lens unit 150 (described later; detachable). The eyepiece 16 is an eyepiece for an eyepiece finder (a peer-in type finder), and the user can view an image displayed on an internal EVF 29 (Electric View Finder; described later) through the eyepiece 16. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether or not the user (photographer) has placed his / her eye on the eyepiece 16. The cover 202 is a cover for a slot that stores a recording medium 200 (described later). The grip unit 90 is a holding unit that is shaped so that the user can easily hold it with the right hand when holding the digital camera 100. The mode change switch 60, the shutter button 61, the main electronic dial 71, the movie button 76, and the like are arranged in positions that can be operated with the index finger of the right hand when the digital camera 100 is held by gripping the grip unit 90 with the little finger, ring finger, and middle finger of the right hand. In the same state, the MC65, touch panel 70a, sub electronic dial 73, four-way key 74, SET button 75, AE lock button 77, enlarge button 78, playback button 79, etc. are arranged in a position that can be operated with the thumb of the right hand.

[0017] <Block diagram of digital camera 100> FIG. 2 is a block diagram showing an example of the configuration of the digital camera 100. The lens unit 150 is a lens unit equipped with an interchangeable photographing lens. The lens 103 is usually composed of multiple lenses, but FIG. 2 shows only one lens for simplicity. The communication terminal 6 is a communication terminal through which the lens unit 150 communicates with the digital camera 100, and the communication terminal 10 is a communication terminal through which the digital camera 100 communicates with the lens unit 150. The lens unit 150 communicates with the system control unit 50 via these communication terminals 6 and 10. The lens unit 150 controls the aperture 1 via the aperture drive circuit 2 by the internal lens system control circuit 4. The lens unit 150 adjusts the focus by displacing the lens 103 via the AF drive circuit 3 by the lens system control circuit 4.

[0018] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50 .

[0019] The imaging unit 22 is an imaging element (imaging sensor) configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 22 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 23 is used to convert an analog signal output from the imaging unit 22 into a digital signal.

[0020] The image processing unit 24 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, EF (flash pre-flash) processing, etc. to be performed. The image processing unit 24 further performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the obtained arithmetic results.

[0021] The memory control unit 15 controls transmission and reception of data between the A / D converter 23, the image processing unit 24, and the memory 32. Output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15. Alternatively, the output data from the A / D converter 23 is written to the memory 32 via the memory control unit 15 without going through the image processing unit 24. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, and image data to be displayed on the display unit 28 and EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0022] The memory 32 also serves as a memory (video memory) for displaying images. The image data for display written in the memory 32 is displayed by the display unit 28 or the EVF 29 via the memory control unit 15. The display unit 28 and the EVF 29 each perform display according to a signal from the memory control unit 15 on a display device such as an LCD or an organic EL. A live view display (LV) can be performed by sequentially transferring and displaying data that has been A / D converted by the A / D converter 23 and stored in the memory 32 to the display unit 28 or the EVF 29. Hereinafter, an image displayed in the live view display is referred to as a live view image (LV image).

[0023] The gaze detection unit 160 detects the user's gaze at the eyepiece unit 16. The gaze detection unit 160 is composed of a dichroic mirror 162, an imaging lens 163, a gaze detection sensor 164, a gaze detection circuit 165, and an infrared light emitting diode 166. Note that since the system control unit 50 can also execute a predetermined process in response to detection of the gaze, the gaze detection unit 160 can also be said to be a part of the operation unit 70.

[0024] In this embodiment, the gaze detection unit 160 detects the gaze by a method called corneal reflex method. The corneal reflex method is a method of detecting the direction and position of the gaze from the positional relationship between the reflected light (particularly the reflected light reflected by the cornea) of the infrared light emitted from the infrared light emitting diode 166 and reflected by the eyeball (eye) 161 and the pupil of the eyeball (eye) 161. It should be noted that the method is not limited to the corneal reflex method, and various other methods for detecting the direction and position of the gaze can be used. For example, a method called scleral reflex method can be used, which utilizes the difference in light reflectance between the iris and the white of the eye.

[0025] The infrared light emitting diode 166 is a light emitting element for detecting the position of the user's line of sight within the viewfinder screen, and irradiates infrared light onto the user's eyeball (eye) 161 placed against the eyepiece 16. The infrared light emitted from the infrared light emitting diode 166 is reflected by the eyeball (eye) 161, and the reflected infrared light reaches the dichroic mirror 162. The dichroic mirror 162 reflects only the infrared light and transmits visible light. The reflected infrared light whose optical path has been changed forms an image on the imaging surface of the line of sight detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical member that constitutes the line of sight detection optical system. The line of sight detection sensor 164 is composed of an imaging device such as a CCD image sensor.

[0026] The gaze detection sensor 164 photoelectrically converts the incident infrared reflected light into an electric signal and outputs it to the gaze detection circuit 165. The gaze detection circuit 165 includes at least one processor, and detects the gaze position of the user from the image or movement of the user's eyeball (eye) 161 based on the output signal from the gaze detection sensor 164, and outputs the detection information to the system control unit 50.

[0027] Various settings of the camera, including the shutter speed and aperture, are displayed on the outside viewfinder display section 43 via an outside viewfinder display section drive circuit 44 .

[0028] The non-volatile memory 56 is an electrically erasable and recordable memory, such as a flash ROM. Constants, programs, and the like for the operation of the system control unit 50 are recorded in the non-volatile memory 56. The programs referred to here include various flow charts to be described later in this embodiment. It is a computer program for executing charts.

[0029] The system control unit 50 is a control unit made up of at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 executes the programs recorded in the nonvolatile memory 56 described above to realize each process of this embodiment described below. The system memory 52 is, for example, a RAM, and the system control unit 50 loads constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, the display unit 28, the EVF 29, and the like.

[0030] The system timer 53 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0031] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, and the like, and detects whether a battery is attached, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage for the necessary period to each unit including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries, an AC adapter, and the like.

[0032] The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0033] The communication unit 54 transmits and receives video signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 54 can also connect to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive image data and various other information from an external device.

[0034] The attitude detection unit 55 detects the attitude of the digital camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 55, it is possible to determine whether the image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. It is also possible to detect the movement of the digital camera 100 (panning, tilting, lifting, whether it is stationary, etc.) using the acceleration sensor or gyro sensor that is the attitude detection unit 55.

[0035] The eyepiece detection unit 57 is an eyepiece detection sensor that detects (approach detection) the approach (eye approach) and removal (eye away) of the eye (object) 161 to the eyepiece unit 16 of the eyepiece viewfinder (finder). The system control unit 50 switches between display (display state) and non-display (non-display state) of the display unit 28 and the EVF 29 depending on the state detected by the eyepiece detection unit 57. More specifically, when at least the digital camera 100 is in a shooting standby state and the switching setting of the display destination of the live view image captured by the imaging unit 22 is set to the automatic switching setting, when the eye is not placed near the camera, the display is turned on as the display unit 28 and the EVF 29 is not displayed. Also, when the eye is placed near the camera, the display is turned on as the display destination of the EVF 29 and the display is not displayed. As the eyepiece detection unit 57 For example, an infrared proximity sensor can be used to detect the approach of an object to the eyepiece 16 of the finder incorporating the EVF 29. When an object approaches, infrared light projected from a light-projecting section (not shown) of the eyepiece detection section 57 is reflected by the object and received by a light-receiving section (not shown) of the infrared proximity sensor. The amount of infrared light received can also determine how close the object is to the eyepiece 16 (eyepiece distance). In this manner, the eyepiece detection section 57 performs eye detection to detect the proximity of the object to the eyepiece 16. In this embodiment, the light-projecting section and light-receiving section of the eyepiece detection section 57 are devices separate from the infrared light-emitting diode 166 and the line-of-sight detection sensor 164 described above. However, the infrared light-emitting diode 166 may also function as the light-projecting section of the eyepiece detection section 57, and the line-of-sight detection sensor 164 may also function as the light-receiving section of the eyepiece detection section 57. In this embodiment, when an object approaching within a predetermined distance from the eyepiece unit 16 is detected from a non-eyepiece state (non-approaching state), it is detected that the eye has been placed in contact with the object. When an object that was detected to be approaching from the eyepiece state (approaching state) moves away from the eyepiece state by a predetermined distance or more, it is detected that the eye has been removed. The threshold value for detecting the eye placement and the threshold value for detecting the removal of the eye may be different, for example, by providing a hysteresis. After detecting the eye placement, the eye placement is maintained in the eye placement state until the removal of the eye is detected. After detecting the removal of the eye, the eye placement is maintained in the non-eye placement state until the eye placement is detected. Note that the infrared proximity sensor is only an example, and other sensors may be used for the eye placement detection unit 57 as long as they can detect the approach of an eye or an object that can be considered as an eye placement.

[0036] The system control unit 50 can detect the following operations or states of the eyepiece unit 16 based on the output from the line-of-sight detection unit 160. The gaze, which was not directed toward the eyepiece 16, is now directed toward the eyepiece 16, i.e., gaze input has begun. A state in which gaze input is being made to the eyepiece 16. A state in which the user is gazing at the eyepiece unit 16 (for example, gaze input in which the amount of movement of the user's gaze position does not exceed a predetermined amount within a predetermined time). The gaze is lifted from the eyepiece 16, i.e., gaze input ends. A state in which no gaze input is being made to the eyepiece 16.

[0037] These operations and states, as well as the position (direction) in which the gaze is directed toward the eyepiece unit 16, are notified to the system control unit 50 via the internal bus, and the system control unit 50 determines what kind of operation (gaze operation) has been performed on the eyepiece unit 16 based on the notified information.

[0038] The operation unit 70 is an input unit that accepts operations from the user (user operations) and is used to input various operational instructions to the system control unit 50. As shown in Fig. 2, the operation unit 70 includes a mode changeover switch 60, a shutter button 61, a power switch 72, a touch panel 70a, and other operation members 70b. The other operation members 70b include the MC 65, a main electronic dial 71, a sub electronic dial 73, a four-way key 74, a SET button 75, a video button 76, an AE lock button 77, a magnification button 78, a playback button 79, a menu button 81, and the like.

[0039] The mode changeover switch 60 switches the operation mode of the system control unit 50 to one of still image shooting mode, video shooting mode, playback mode, etc. Modes included in the still image shooting mode include auto shooting mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide shooting settings for different shooting scenes. The mode changeover switch 60 allows the user to directly switch to one of these modes. Alternatively, after switching once to a list screen of shooting modes with the mode changeover switch 60, the user may selectively switch to one of the multiple displayed modes using other operating members. Similarly, there are multiple video shooting modes. It may be included.

[0040] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on when the shutter button 61 is pressed halfway (instruction to prepare for shooting) and generates a first shutter switch signal SW1. The system control unit 50 starts preparation operations for shooting such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing in response to the first shutter switch signal SW1. The second shutter switch 64 is turned on when the shutter button 61 is pressed fully (instruction to shoot) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of operations for shooting processing, from reading out a signal from the imaging unit 22 to writing a captured image to the recording medium 200 as an image file, in response to the second shutter switch signal SW2.

[0041] The touch panel 70a and the display unit 28 can be configured as one unit. For example, the touch panel 70a is configured so that the light transmittance does not interfere with the display of the display unit 28, and is attached to the upper layer of the display surface of the display unit 28. Then, input coordinates on the touch panel 70a are associated with display coordinates on the display surface of the display unit 28. This makes it possible to provide a GUI (Graphical User Interface) that makes it seem as if the user can directly operate the screen displayed on the display unit 28. The system control unit 50 can detect the following operations or states on the touch panel 70a.

[0042] A finger or pen that has not been touching the touch panel 70a touches the touch panel 70a again, that is, the start of touching (hereinafter referred to as touch-down). A state in which the touch panel 70a is touched with a finger or a pen (hereinafter, referred to as Touch-On). A finger or a pen is moved while touching the touch panel 70a (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 70a is removed (released) from the touch panel 70a, that is, the touch ends (hereinafter, referred to as "touch-up"). A state in which nothing is being touched on the touch panel 70a (hereinafter referred to as Touch-Off).

[0043] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0044] These operations and states, as well as the position coordinates of the touch panel 70a touched by a finger or pen, are notified to the system control unit 50 via the internal bus. The system control unit 50 then determines what kind of operation (touch operation) has been performed on the touch panel 70a based on the notified information. For touch moves, the direction of movement of a finger or pen moving on the touch panel 70a can also be determined for each vertical and horizontal component on the touch panel 70a based on changes in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 70a, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced across the touch panel 70a as if flicking it. If a touch move of a predetermined distance or more at a predetermined speed or more is detected, a touch up is performed immediately. When this is detected, it can be determined that a flick has been performed (it can be determined that a flick has been performed following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer to each other is called a pinch in, and a touch operation in which the touch positions are moved away from each other is called a pinch out. Pinch out and pinch in are collectively called a pinch operation (or simply a pinch). The touch panel 70a may be any of various touch panel types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. There are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of a finger or a pen to the touch panel, and either type may be used.

[0045] The system control unit 50 can set the method of specifying an index position (such as an AF position (position of the AF frame)) by a touch operation to absolute position specification or relative position specification, depending on a user operation on the operation unit 70. For example, in the case of absolute position specification, when the touch panel 70a is touched, the system control unit 50 sets an index position associated with the touch position. That is, the position coordinates (input coordinates) where the touch operation is performed are associated with the position coordinates (display coordinates) of the display unit 28. On the other hand, in the case of relative position specification, the position coordinates where the touch operation is performed are not associated with the position coordinates of the display unit 28. In the case of relative position specification, the system control unit 50 moves the index position from the currently set index position in the direction of movement of the touch move by a distance corresponding to the amount of movement of the touch move, regardless of the position of touch down on the touch panel 70a.

[0046] <Operation of the digital camera 100> 3 and 4 are flow charts showing details of the shooting mode process performed by the digital camera 100. This process is realized by the system control unit 50 expanding a program recorded in the non-volatile memory 56 into the system memory 52 and executing it. For example, when the digital camera 100 is started in the shooting mode or the system control unit 50 switches to the shooting mode, the system control unit 50 initializes flags, control variables, etc., and then displays a live view image captured by the imaging unit 22 on the display unit 28 and EVF 29. In addition, the system control unit 50 displays an information icon indicating the camera setting value based on the camera setting value, superimposed on the live view image. After that, the system control unit 50 starts the shooting mode process of FIGS. 3 and 4.

[0047] 3, the system control unit 50 determines whether or not an operation to switch between enabling and disabling the gaze detection function has been performed on the operation unit 70. If an operation has been performed, the process proceeds to S302, and if not, the process proceeds to S303.

[0048] In S302, the system control unit 50 changes the camera setting value so as to switch the line of sight detection function between enabled and disabled in response to the operation in S301.

[0049] In S303, the system control unit 50 determines whether an operation to switch between enabling and disabling the gaze pointer display has been performed on the operation unit 70. If an operation has been performed, the process proceeds to S304, and if not, the process proceeds to S305. The gaze pointer is a display item (indicator) that indicates the gaze position (position where gaze input has occurred) on the eyepiece unit 16 (display surface of the EVF 29), and is displayed at the gaze position.

[0050] In S304, the system control unit 50 changes the camera setting value so as to switch between enabling and disabling the gaze pointer display in response to the operation in S303.

[0051] In S305, the system control unit 50 switches between enable / disable of the gaze position determination function for determining the gaze position in response to pressing (ON) of the first shutter switch 62 for the operation unit 70. If an operation has been performed, the process proceeds to S306, and if not, the process proceeds to S307.

[0052] In S306, the system control unit 50 changes the camera setting value so as to switch between enable / disable of the gaze position determination function in response to the operation in S305.

[0053] In S307, the system control unit 50 determines whether or not an operation to switch the AF method setting has been performed on the operation unit 70. If an operation has been performed, the process proceeds to S308, and if not, the process proceeds to S309.

[0054] In S308, the system control unit 50 changes the camera setting values ​​so as to switch the AF method setting in response to the operation in S307.

[0055] In S309, the system control unit 50 determines whether or not the gaze detection function is enabled. If it is enabled, the process proceeds to S310, and if it is not enabled, the process proceeds to S312 (FIG. 4).

[0056] In S310, the system control unit 50 determines whether or not the gaze pointer display is enabled. If it is enabled, the process proceeds to S311, and if it is not enabled, the process proceeds to S312 (FIG. 4).

[0057] In S311, the system control unit 50 acquires information on the gaze position from the gaze detection unit 160, and displays a gaze pointer on the display surface of the EVF 29 at the gaze position.

[0058] In S312, the system control unit 50 acquires the current lens ID (identification information (individual information) that identifies the lens unit 150) and the currently set aperture value. For example, the system control unit 50 acquires the lens ID and the aperture value from the lens unit 150. The system control unit 50 may acquire the lens ID and the aperture value from the lens unit 150 and record the lens ID and the aperture value in the system memory 52 in advance. Then, in S312, the system control unit 50 may acquire the lens ID and the aperture value from the system memory 52.

[0059] In S313, the system control unit 50 determines whether or not the AF target fixing setting is enabled. The AF target fixing setting is a setting (function) that fixes the AF target (subject to be focused on; subject to be focused on). If enabled, the process proceeds to S314, and if not, the process proceeds to S315. The AF target can also be said to be a focus detection area for focusing on the corresponding subject.

[0060] The system control unit 50 can switch between enable / disable of the AF target fixing setting in response to an operation (operation input) on the operation unit 70. For example, the AF target fixing setting is switched between enable / disable in response to an operation on an operation member that can be operated while pressing the shutter button 61, such as the AF-ON button, the AEL button, the AF-SEL button, a touch panel, a multi-controller, an OTP, or a stop-down button. The operation member that enables the AF target fixing setting and the operation member that disables the AF target fixing setting may or may not be different from each other. The system control unit 50 may toggle between enable / disable of the AF target fixing setting in response to an operation on a specific operation member (one operation member). Specifically, if the AF target fixing setting is disabled, the system control unit 50 may enable the AF target fixing setting in response to an operation on the specific operation member, and if the AF target fixing setting is enabled, the system control unit 50 may disable the AF target fixing setting in response to an operation on the specific operation member. The user can also switch between enable / disable of the AF target fixing setting from a menu screen.

[0061] In this embodiment, when the AF target fixing setting is disabled, the system control unit 50 The system controller 50 determines the subject displayed at the gaze position (the position where the gaze input was made) in the menu image as the AF target, and controls to focus on the determined subject. Therefore, if the gaze position changes to view a subject different from the subject set as the AF target, the AF target is switched to the subject at the gaze position. Then, when the system controller 50 switches the AF target fixing setting from disabled to enabled, the system controller 50 controls to fix the AF target to the subject that was focused on (the AF target before the AF target fixing setting was switched) and continue focusing on that subject. When the AF target fixing setting is enabled, the AF target is not switched even if the gaze position changes to view a subject different from the subject set as the AF target.

[0062] In S314, the system control unit 50 fixes the AF target regardless of the gaze position. This allows the AF target to be fixed even if the gaze is shifted to a point other than the AF target, making it possible to check the framing (composition), and enables operations that match the user's intentions.

[0063] In S315, the system control unit 50 determines the AF target according to the gaze position. When the gaze position is changed to view a subject different from the subject set as the AF target, the system control unit 50 switches the AF target to the subject at the gaze position.

[0064] Depending on the lens unit 150 (lens type, aperture, etc.), the brightness of the live view image may decrease from the center to the edge of the live view image (vignetting). In S316, the system control unit 50 calculates the amount of brightness decrease due to vignetting (the amount of brightness decrease of the image from the center to the edge) based on the lens ID and aperture value acquired in S312, and determines whether the amount of brightness decrease is equal to or greater than a predetermined threshold. Note that the amount of brightness decrease may be calculated in S312. If the amount of brightness decrease is equal to or greater than the threshold, the process proceeds to S317, and if not (if the amount of brightness decrease is less than the threshold), the process proceeds to S313.

[0065] In S317, the system control unit 50 detects the area of ​​the subject present at the gaze position from the live view image using the image processing unit 24. For example, the system control unit 50 divides the live view image into a plurality of predetermined areas, and calculates the similarity between each area and the area of ​​the gaze position based on the color and brightness information of each area. Then, the system control unit 50 determines the area consisting of areas with a similarity higher than a predetermined threshold as the area of ​​the subject present at the gaze position.

[0066] In S318, the system control unit 50 uses the image processing unit 24 to generate a mask that distinguishes the area (gaze area) detected in S317 from other areas. The mask is used to process only the gaze area and disable processing on other areas (areas different from the gaze area). In this embodiment, a bitmap image in which the pixel value of the gaze area is 1 and the pixel value of the other areas is 0 is generated as the mask.

[0067] The gaze area is not limited to the area of ​​the subject present at the gaze position. For example, the gaze area may be an area including the subject present at the gaze position and larger than the subject. The gaze area may be another area including the gaze position, such as an area consisting of a plurality of pixels whose distance from the gaze position is less than a predetermined distance (a distance of a predetermined number of pixels). The size of the gaze area may be a predetermined fixed size, or may be determined or changed depending on the display size of the subject present at the gaze position (size on the live view image). For example, the gaze area may be larger as the display size of the subject present at the gaze position is larger. The shape of the gaze area is not particularly limited, and may be a circle such as a perfect circle or an ellipse, or a polygon such as a triangle or a hexagon.

[0068] In S319, the system control unit 50 uses the image processing unit 24 to extract a gaze area (area based on the gaze position) from the live view image, the gaze area being specified by the mask generated in S318. In this embodiment, the system control unit 50 changes (increases) the luminance of the gaze area by an amount of change based on the amount of luminance reduction due to peripheral shading (calculated in S316; amount of light according to image height). The luminance of the gaze area (pixel) is changed, for example, by multiplying the pixel value by a gain value or adding an offset value to the pixel value. As a result, for example, when the luminance of the gaze area is lower than a threshold, the luminance of the gaze area is corrected to the luminance of the threshold. The luminance of other areas (areas different from the gaze area) is not changed. Note that there is no particular limitation on the method of changing the luminance of the gaze area. For example, the luminance of each pixel in the gaze area may be raised or lowered uniformly, or may be brought closer to a predetermined luminance.

[0069] In this embodiment, the luminance of the gaze area (area based on the gaze position) is controlled by image processing of the displayed image, but local luminance control (partial control of display luminance) by the display device itself can also be achieved. For example, each of the display unit 28 and the EVF 29 may locally control the display luminance based on a size signal of the gaze area on a display such as an LCD or an organic EL.

[0070] In addition, in this embodiment, the brightness of the image in the gaze area is controlled, but the object of the present invention can be achieved by performing other partial image processing on the image in the gaze area. For example, partial image processing includes white balance, color conversion, enlargement processing, and sharpening processing. If the image processing parameters of the gaze area are changed, the image processing itself may be performed on the entire image. The system control unit 50 uses the image processing unit 24 to perform partial image processing on the gaze area (area based on the gaze position) of the live view image specified by the mask generated in S318, and displays the processed images on the EVF 29 in sequence as live view images.

[0071] 3 and 4, after the shooting mode process starts, the system control unit 50 reads out the signal from the imaging unit 22 by the second shutter switch signal SW2, and writes the captured image as an image file to the recording medium 200. At this time, the brightness control (partial image processing) for the line of sight area (pixels) described above is not performed on the image to be recorded. Alternatively, it is also possible to configure the system so that both an image that has been subjected to partial image processing and an image that has not been subjected to partial image processing are recorded as the recorded image.

[0072] As described above, according to this embodiment, focus control by gaze input (control for focusing on a subject according to the gaze position) is performed when the AF target fixing setting is disabled, and is not performed when the AF target fixing setting is enabled. On the other hand, brightness control by gaze input (control of brightness of the gaze area) is performed regardless of whether the AF target fixing setting is enabled. This makes it possible to perform focus control and brightness control by gaze input in an optimal manner (with good operability). For example, when only brightness control by gaze input is desired, it is possible to prevent focus control by gaze input from being performed against the user's intention.

[0073] In addition, in this embodiment, the focus control and brightness control by eye gaze input have been described in detail, but the settings of the imaging parameters may be changed by selecting an information icon displayed in a live view image by eye gaze input and operating an electronic dial or button. For example, as shown in FIG. 6, when the user is gazing at an information icon such as shutter speed or aperture displayed in a live view image showing camera setting values, the system control unit 50 causes the EVF 29 to select the information icon being gazed at (highlighted, etc.). In this state, the user may turn, for example, an electronic dial to change the setting value of the imaging parameter corresponding to the information icon, such as shutter speed, aperture, or focal length.

[0074] The various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0075] In addition, although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0076] For example, an example has been described in which the luminance of the gaze area is changed when the condition that the amount of luminance reduction of peripheral light is equal to or greater than a predetermined threshold is satisfied, but the luminance of the gaze area may be changed regardless of whether the condition is satisfied. That is, S316 in FIG. 4 may be omitted, and the processes of S317 to S319 may always be performed. Other conditions may be used as a condition for switching whether to change the luminance of the gaze area. In addition, an example has been described in which the subject of the AF target is fixed when the AF target is not determined according to the gaze position, but the AF position (position to be focused) may be fixed when the AF target is not determined according to the gaze position. In addition, an example has been described in which the enable / disable of various settings such as the AF target fixing setting is switched according to a user operation, but the enable / disable of various settings may be automatically switched according to the shooting scene, etc.

[0077] As another AF control method, the AF target setting may be switched by a user's operation instead of always reflecting the gaze position as in this embodiment. For example, as shown in Fig. 7, when the shutter switch signal SW1 (or an operation from another independent operating member) is input, the AF frame may move to the subject displayed at the gaze position and the subject may be determined as the AF target.

[0078] Instead of the process in Fig. 4, the process in Fig. 5 may be performed. According to the process in Fig. 5, the focus control by the eye gaze input and the brightness control by the eye gaze input are selectively performed, which further improves operability and convenience.

[0079] In S320, the system control unit 50 determines whether or not the AF target fixing setting is valid. If it is valid, the process proceeds to S321, and if it is not valid, the process proceeds to S322.

[0080] In S321, the system control unit 50 fixes the AF target regardless of the gaze position.

[0081] In S322, the system control unit 50 determines the AF target according to the gaze position.

[0082] In S323, the system control unit 50 determines whether or not the brightness correction setting is enabled. The brightness correction setting is a setting that changes (corrects) the brightness of the line of sight area. If it is enabled, the process proceeds to S324, and if it is not enabled, the process proceeds to S326. As with the AF target fixing setting, the system control unit 50 can switch between enabling / disabling the brightness correction setting in response to an operation on the operation unit 70. However, the operation member that switches between enabling / disabling the brightness correction setting is different from the operation member that switches between enabling / disabling the AF target fixing setting.

[0083] In S324, the system control unit 50 generates a mask for distinguishing the gaze region from other regions, using the image processing unit 24. For example, a region consisting of a plurality of pixels whose distance from the gaze position is a predetermined distance (a distance corresponding to a predetermined number of pixels) or less is regarded as the gaze region, and a bitmap image in which the pixel value of the gaze region is 1 and the pixel value of other regions is 0 is generated as the mask.

[0084] In S325, the system control unit 50 uses the image processing unit 24 to change the luminance of the gaze area specified by the mask generated in S324 in the live view image. For example, the system control unit 50 increases the luminance of an area in the gaze area that is lower than a predetermined luminance to the predetermined luminance. The luminance of other areas (areas different from the gaze area) is not changed.

[0085] In S326, the system control unit 50 uses the image processing unit 24 to perform luminance correction of the line of sight area. As a result, the luminance of the line of sight area is restored to the luminance before being changed in S325.

[0086] In the above-mentioned embodiment, the present invention is applied to a digital camera (imaging device), but the present invention is not limited to this example and can be applied to any electronic device that can accept gaze input. For example, the present invention can be applied to a personal computer, a PDA, a mobile phone terminal, a portable image viewer, a printer device, a digital photo frame, a music player, a game machine, an electronic book reader, a video player, and the like. The present invention can also be applied to a wearable device such as a head mounted display (HMD), a display device (including a projection device), a tablet terminal, a smartphone, an AI speaker, a home appliance device, an in-vehicle device, a medical device, and the like. The process of accepting gaze input may be a process of detecting a gaze, or a process of acquiring information on a gaze detected by an external device. The electronic device to which the present invention is applied may acquire an image captured by a predetermined imaging control, and the imaging itself may be performed by an external device.

[0087] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0088] 100: Digital camera 50: System control unit 160: Line-of-sight detection unit

Claims

1. a gaze input means for receiving a gaze input by a gaze of a user looking at the display means; An acquisition means for acquiring an image captured under a predetermined imaging control; a display control means for displaying the image acquired by the acquisition means on the display means; a first operation member that accepts a first operation input by a user's hand; a second operation member that receives a second operation input different from the first operation input by a user's hand; The image is displayed on the display means by the display control means, and a gaze input is made to a part of the image, When the first operation input is received, the predetermined imaging control is set based on the part of the image; When the second operation input is received, image processing is performed on the part of the image. A control means; 1. An electronic device comprising:

2. The predetermined imaging control setting is a setting of a focus detection area for focusing on a subject corresponding to the portion of the image.

2. The electronic device according to claim 1 .

3. The predetermined imaging control settings are imaging parameter settings corresponding to an icon displayed on the portion of the image.

2. The electronic device according to claim 1 .

4. The imaging parameters include at least one of an aperture value, a shutter speed, and a focal length.

4. The electronic device according to claim 3.

5. The image processing is a process for increasing the luminance of the part of the image displayed on the display means.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

6. When the luminance of the portion is lower than a threshold value, the control means increases the luminance of the portion to the threshold value.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

7. The image processing is a process of changing an image processing parameter of the part of the image displayed on the display means.

7. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

8. The image processing is processing of the part of the image displayed on the display means.

8. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

9. The control means when the second operation input is received, image processing is performed on the part of the image displayed on the display means; The image is recorded on the recording medium without undergoing the image processing.

9. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

10. The display control means displays an indicator corresponding to the portion of the image.

10. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

11. Receiving gaze input by a user looking at a display means; acquiring an image captured under a predetermined imaging control; displaying the acquired image on said display means; receiving a first operation input by a user's hand; receiving a second operation input different from the first operation input by a user's hand; The image is displayed on the display means, and a gaze input is made to a part of the image, When the first operation input is received, the predetermined imaging control is set based on the part of the image; When the second operation input is received, image processing is performed on the part of the image. Steps and 13. A method for controlling an electronic device comprising:

12. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 10.

13. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 10.

Images