Electronic apparatus and method for controlling the same

Abstract

To generate calibration data with higher accuracy.SOLUTION: An electronic apparatus has: detection means that detects a user's line-of-sight on a screen; and control means that performs control to execute calibration on the basis of information on the user's line-of-sight detected by the detection means. When a predetermined condition related to the calibration is satisfied, the control means performs control to perform notification related to the calibration.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an electronic device that allows gaze input based on a user's gaze, and a method for controlling the electronic device. [Background technology]

[0002] Conventionally, there is known a camera that detects the line of sight of a user (photographer), detects which position (area) in the viewfinder the photographer is looking at, and controls a photographing function such as automatic focus adjustment. However, the accuracy of detecting the user's line of sight depends on the pupil diameter of the user's eye, the way the user looks into the viewfinder, and the brightness of the surroundings. Therefore, it is necessary to perform calibration to obtain the user's line of sight as data and to obtain a calibration value for calculating the position of the user's line of sight. By performing calibration, the difference between the position the user is looking at and the calculated line of sight position corresponding to the user's line of sight becomes small, and the user can specify a position or execute a function using the line of sight without feeling uncomfortable. Patent Document 1 discloses that the optical device detects whether it is in a horizontal or vertical position at the start of calibration, and stores a calibration value calculated from the detected position and the user's line of sight information (line of sight data) as calibration data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 07-255676 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, when acquiring gaze data, which is a plurality of gaze information for creating calibration data, only the orientation of the optical device at the start of calibration is acquired. The orientation of the optical device from the second point onwards at the timing of acquiring gaze data for each of the plurality of detection points is not taken into consideration. If the orientation of the optical device at the timing of acquiring gaze data for the first point (index 1) differs from the orientation of the optical device at the timing of acquiring gaze data for the second point (index 2), the accuracy of the gaze data decreases. This also affects the calibration data including the calculated calibration value. This causes a discrepancy between the position where the user is looking and the gaze position corresponding to the user's gaze.

[0005] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to make it possible to generate calibration data with higher accuracy. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the present invention has a detection means for detecting a user's line of sight toward a screen, and a control means for controlling calibration to be performed based on information regarding the user's line of sight detected by the detection means, and is characterized in that the control means controls to issue a notification regarding the calibration when predetermined conditions regarding the calibration are satisfied. Effect of the Invention

[0007] According to the present invention, it is possible to generate calibration data with higher accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of a digital camera 100. [Diagram 2] FIG. 1 is a block diagram showing a configuration of a digital camera 100. [Diagram 3] 11 is a control flowchart of a calibration process. [Figure 4]11 is a control flowchart of a data acquisition process in the calibration process. [Diagram 5] 13 is a display example when gaze data, which is gaze information, is acquired. [Figure 6] 1 is an explanatory diagram of the attitude of digital camera 100 during data acquisition processing and the positional relationship between gaze detection block 160 and eye 161. FIG. [Figure 7] 13 is a display example of a setting screen related to a line of sight. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] FIG. 1 shows an external view of a digital camera 100 as an example of a device (electronic device) to which the present invention can be applied. FIG. 1(a) is a front perspective view of the digital camera 100, and FIG. 1(b) is a rear perspective view of the digital camera 100. In FIG. 1, a display unit 28 is provided on the rear surface of the camera and displays images and various information. A touch panel 70a is a touch operation member and can detect touch operations on the display surface (operation surface) of the display unit 28. An outside-finder liquid crystal display unit 43 is a display unit provided on the top surface of the camera and displays various settings of the camera, including shutter speed and aperture.

[0010] The shutter button 61 is an operation unit for issuing a shooting instruction. The mode change switch 60 is an operation unit for switching between various modes. The terminal cover 40 is a cover for protecting a connector (not shown) that connects a connection cable with an external device to the digital camera 100. The main electronic dial 71 is a rotary operation member included in the operation unit 70, and by turning this main electronic dial 71, settings such as shutter speed and aperture can be changed. The power switch 72 is an operation member for switching the power of the digital camera 100 ON and OFF. The power switch 72 is included in the operation unit 70. The sub electronic dial 73 is a rotary operation member included in the operation unit 70, and can move the selection frame, advance images, and the like. The cross key 74 is an operation member included in the operation unit 70 and has a push button that can be pressed in four directions. Operations can be performed according to the direction in which the cross key 74 is pressed. The SET button 75 is included in the operation unit 70, is a push button, and is mainly used to confirm a selection item, and the like. The video button 76 is an operating member used to instruct starting and stopping video shooting (recording), and is included in the operation unit 70.

[0011] The M-Fn button 77 is included in the operation unit 70, and is used to quickly change various settings such as white balance and ISO sensitivity by pressing it in a shooting standby state. The user can arbitrarily assign various functions to the M-Fn button 77, not only in the shooting standby state. It is also possible to assign a function related to gaze input, and for example, in the shooting standby state, the gaze input function can be switched ON / OFF by pressing the M-Fn button 77. It is also used when performing calibration, which is performed to more accurately detect the position where the user is looking using the gaze detection block 160 described later. When the user gazes at the gaze point, which is the displayed calibration display, and presses the M-Fn button 77, it functions as a confirmation operation. Calibration will be described later.

[0012] The playback button 79 is included in the operation unit 70 and is an operation button for switching between shooting mode and playback mode. Pressing the playback button 79 during shooting mode switches to playback mode, and the most recent image of the images recorded on the recording medium 200 can be displayed on the display unit 28. The menu button 81 is included in the operation unit 70 and pressing it displays a menu screen on the display unit 28 where various settings can be made. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the cross key 74, and the SET button 75.

[0013] The communication terminal 10 is a communication terminal for the digital camera 100 to communicate with a lens unit 150 (detachable) described later. The eyepiece 16 is an eyepiece of an eyepiece finder (a peer-in type finder), and the user can view an image displayed on an EVF (Electric View Finder) 29 inside the finder through the eyepiece 16. If an optical image of a subject (subject image) can be obtained through the lens unit 150, the optical image (subject image) can be viewed through the eyepiece 16 by the internal optical finder. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether or not the photographer has placed his / her eye on the eyepiece 16. The lid 202 is a lid of a slot that stores a recording medium 200. The grip unit 90 is a holding unit shaped to be easily gripped with the right hand when the user holds the digital camera 100. The shutter button 61 and main electronic dial 71 are located at positions that can be operated by the index finger of the right hand when the digital camera is held by gripping the grip portion 90 with the little finger, ring finger, and middle finger of the right hand. In the same state, the sub electronic dial 73 is located at a position that can be operated by the thumb of the right hand.

[0014] FIG. 2 is a block diagram showing an example of the configuration of a digital camera 100 and a lens unit 150, which are one of the electronic devices according to this embodiment. In FIG. 2, the lens unit 150 is a lens unit equipped with an interchangeable photographing lens, and is a lens for obtaining a subject image. The lens 103 is usually composed of multiple lenses, but is shown here as only one lens for simplicity. The communication terminal 6 is a communication terminal for the lens unit 150 to communicate with the digital camera 100. The lens unit 150 communicates with the system control unit 50 via this communication terminal 6 and the above-mentioned communication terminal 10. The lens system control circuit 4 controls the aperture 1 via the aperture drive circuit 2. The lens system control circuit 4 adjusts the focus by displacing the lens 103 via the AF drive circuit 3.

[0015] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50 .

[0016] The imaging unit 22 is an imaging element that converts an optical image into an electrical signal and is composed of a CCD, a CMOS element, etc. The A / D converter 23 is used to convert an analog signal output from the imaging unit 22 into a digital signal.

[0017] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the data from the A / D converter 23 or data from the memory control unit 15 described later. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing to be performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the arithmetic results obtained.

[0018] The memory control unit 15 controls data transmission and reception between the A / D converter 23, the image processing unit 24, and the memory 32. Output data from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, and image data to be displayed on the display unit 28 and the EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0019] The memory 32 also serves as a memory (video memory) for displaying images. The image data for display written in the memory 32 is displayed by the display unit 28 and the EVF 29 via the memory control unit 15. The display unit 28 and the EVF 29 perform display on a display device such as an LCD or an organic EL in response to a signal from the memory control unit 15. A live view display (LV display) can be performed by sequentially transferring and displaying data that has been A / D converted by the A / D converter 23 and stored in the memory 32 to the display unit 28 or the EVF 29. Hereinafter, an image displayed in live view will be referred to as a live view image (LV image).

[0020] The system timer 53 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0021] The mode changeover switch 60, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operation means for inputting various operation instructions to the system control unit 50. The mode changeover switch 60 switches the operation mode of the system control unit 50 to one of a still image shooting mode, a video shooting mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene discrimination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, there are various scene modes and a custom mode that are shooting settings according to shooting scenes. The mode changeover switch 60 allows the user to directly switch to one of these modes. Alternatively, a list screen of shooting modes may be displayed with the mode changeover switch 60, one of the displayed multiple modes may be selected, and the mode may be switched using other operation members. Similarly, the video shooting mode may also include multiple modes.

[0022] The first shutter switch 62 is turned on and generates a first shutter switch signal SW1 when the shutter button 61 provided on the digital camera 100 is pressed halfway (instruction to prepare for shooting) during operation of the shutter button 61. The first shutter switch signal SW1 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0023] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file, in response to the second shutter switch signal SW2.

[0024] The operation unit 70 is a variety of operation members that serve as an input unit for receiving operations from the user. The operation unit 70 includes at least the following operation parts: the shutter button 61, a touch panel 70a, a main electronic dial 71, a power switch 72, a sub electronic dial 73, a cross key 74, a SET button 75, a movie button 76, an M-Fn button 77, a playback button 79, and a menu button 81.

[0025] The touch panel 70a and the display unit 28 can be configured as one unit. For example, the touch panel 70a is configured so that the light transmittance does not interfere with the display of the display unit 28, and is attached to the upper layer of the display surface of the display unit 28. Then, input coordinates on the touch panel 70a are associated with display coordinates on the display screen of the display unit 28. This makes it possible to provide a GUI (Graphical User Interface) that allows the user to directly operate the screen displayed on the display unit 28. The system control unit 50 can detect the following operations or states on the touch panel 70a. A finger or pen that has not been touching the touch panel 70a touches the touch panel 70a again, that is, the start of touching (hereinafter referred to as touch-down). The touch panel 70a is in a state of being touched with a finger or a pen (hereinafter, referred to as Touch-On). Touching the touch panel 70a with a finger or a pen and moving it (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 70a is released, that is, the touch ends (hereinafter, referred to as "touch-up"). A state in which nothing is being touched on the touch panel 70a (hereinafter referred to as Touch-Off).

[0026] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. Touch-move is also detected while touch-on is detected. Even if touch-on is detected, touch-move will not be detected if the touch position has not moved. After it is detected that all fingers or pens that were touching have touched up, touch-off occurs.

[0027] These operations and states, as well as the position coordinates of the finger or pen touching the touch panel 70a, are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 70a based on the notified information. For touch moves, the direction of movement of the finger or pen moving on the touch panel 70a can also be determined for each vertical and horizontal component on the touch panel 70a based on the change in the position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 70a as if flicking it. If a touch move of a predetermined distance or more at a predetermined speed or more is detected and a touch up is detected immediately, it can be determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched simultaneously and the touch positions are brought closer together is called pinch in, and a touch operation in which the touch positions are moved away from each other is called pinch out. Pinch out and pinch in are collectively called pinch operation (or simply pinch). The touch panel 70a may be of any of various touch panel types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. Depending on the type, there are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of a finger or a pen to the touch panel, and either type may be used.

[0028] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, etc., and detects whether a battery is attached, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage for the necessary period to each unit including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries, an AC adapter, etc.

[0029] The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0030] The communication unit 54 is connected wirelessly or via a wired cable, and transmits and receives video signals and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including live view images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive images and various other information from external devices.

[0031] The non-volatile memory 56 is an electrically erasable and recordable memory, and may be, for example, a Flash-ROM. Constants, programs, and the like for the operation of the system control unit 50 are stored in the non-volatile memory 56. The programs referred to here are computer programs for executing various flowcharts described later in this embodiment.

[0032] The system control unit 50 is a control unit consisting of at least one processor or circuit, and controls the entire digital camera 100. Each process of this embodiment, which will be described later, is realized by executing a program recorded in the nonvolatile memory 56 described above. The system memory 52 may be, for example, a RAM, in which constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are loaded. The system control unit 50 also performs display control by controlling the memory 32, the display unit 28, and the like.

[0033] The attitude detection unit 55 detects attitude information (orientation) of the housing of the digital camera 100 with respect to the direction of gravity. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. It is also possible to detect the movement of the digital camera 100 (panning, tilting, lifting, whether or not it is stationary, etc.) using the acceleration sensor or gyro sensor that is the attitude detection unit 55. Based on the attitude detected by the attitude detection unit 55, it is possible to determine whether the image captured by the imaging unit 22 is an image captured with the digital camera 100 held horizontally or held vertically. The system control unit 50 can add attitude information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image.

[0034] When performing calibration, which will be described later, it is important to understand the positional relationship between digital camera 100 and eye 161. Therefore, when acquiring user's gaze data, which is gaze information, by calibration, the orientation of digital camera 100 detected by orientation detection unit 55 is also acquired and saved as orientation information. This makes it possible to determine the gaze position corresponding to the gaze more accurately, even when the user takes pictures at various angles.

[0035] The gaze detection block 160 detects (accepts) the gaze of the user's eyeball (eye) 161 placed in the eyepiece unit 16. The gaze detection block 160 is composed of a dichroic mirror 162, an imaging lens 163, a gaze detection sensor 164, an infrared light emitting diode 166, and a gaze detection circuit 165.

[0036] The infrared light emitting diode 166 is a light emitting element for detecting the user's line of sight on the display unit within the viewfinder, and irradiates infrared light onto the user's eyeball (eye) 161 placed against the eyepiece unit 16. The infrared light emitted from the infrared light emitting diode 166 is reflected by the eyeball (eye) 161, and the reflected infrared light reaches the dichroic mirror 162. The dichroic mirror 162 reflects only the infrared light and transmits visible light. The reflected infrared light, whose optical path has been changed, forms an image on the imaging surface of the line of sight detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical member that constitutes the line of sight detection optical system. The line of sight detection sensor 164 is composed of an imaging device such as a CCD image sensor.

[0037] The gaze detection sensor 164 photoelectrically converts the incident infrared reflected light into an electric signal and outputs it to the gaze detection circuit 165. The gaze detection circuit 165 includes at least one processor, and detects the user's gaze from the image or movement of the user's eyeball (eye) 161 based on the output signal from the gaze detection sensor 164, and outputs the detection information to the system control unit 50.

[0038] In the present invention, the gaze detection block 160 is used to detect the gaze by a method called the corneal reflex method. The corneal reflex method is a method for detecting the direction and position of the gaze from the positional relationship between the light reflected by the cornea of ​​the eyeball (eye) 161 and the pupil of the eyeball (eye) 161, and infrared light emitted from the infrared light emitting diode 166. There are also various other methods for detecting the direction and position of the gaze, such as a method called the scleral reflex method that utilizes the difference in light reflectance between the black and white of the eye. Note that a gaze detection means other than the above may be used as long as it can detect the direction and position of the gaze.

[0039] The user's line of sight detected by the line of sight detection block 160 is used as a function to display an AF frame at the line of sight position corresponding to the user's line of sight in the digital camera 100, for example, and to execute AF. The AF frame moves following the movement of the user's line of sight, and AF can always be executed at the line of sight position corresponding to the line of sight. It is applicable not only to AF, but also to AE, AWB, and the like. Also, by accumulating which subject the user looked at and how long it took during shooting and saving it together with the image, various functions can be proposed according to the line of sight position corresponding to the user's line of sight and the accumulated time during playback, etc. Also, not only the digital camera 100 but also HMDs (head mounted displays) have high resolution around the line of sight position corresponding to the line of sight, and low resolution as it moves away from the line of sight position corresponding to the line of sight. Some AR (augmented reality) glasses display detailed information about buildings and advertisements in the direction the user is looking.

[0040] Calibration is a calibration step for detecting the gaze of a user using the gaze detection block 160 and determining a gaze position corresponding to the gaze of the user more accurately. The gaze detection block 160 can detect the gaze of a user without performing calibration and determine a gaze position corresponding to the gaze. However, the structure of the entire human eye, including the eyelids, varies from person to person, and it may be difficult for some users to determine a gaze position corresponding to the gaze. By performing calibration, it is possible to obtain gaze data, which is gaze information specific to the user who uses the digital camera 100. By calculating a calibration value from the acquired user-specific gaze data, it is possible to more accurately determine a gaze position corresponding to the gaze input by the user. In addition, by performing calibration multiple times, it is possible to obtain a more accurate calibration value. By performing calibration in various situations, such as under bright natural light, under a fluorescent light, in a dark place, and with glasses on, it is possible to obtain a calibration value corresponding to various situations. When gaze detection is expected to be performed in various postures, as in the digital camera 100 of this embodiment, the positional relationship between the EVF 29 of the digital camera 100 and the eye 161 changes, and the way of looking into the EVF 29 may change. If the positional relationship between the gaze detection block 160 and the eyes 161 changes during calibration, the gaze data, which is the acquired gaze information, changes significantly, causing a deviation in the calibration value. Therefore, it is desirable to perform calibration in each posture, and it is undesirable to change the posture during calibration.

[0041] 6(a) to 6(c) show the orientation of digital camera 100 and the relative positional relationship between gaze detection block 160 and eye 161. Fig. 6(a) is a diagram in which the orientation of digital camera 100 is classified into four types. Here, the orientation of the body of digital camera 100 is classified into the following four orientations (positions). When the eyepiece 16 is located within the range of area 611 (horizontal normal position). When the eyepiece 16 is located within the range of area 612 (left vertical position). When the eyepiece 16 is located within the range of area 613 (right vertical position). When the eyepiece 16 is located within the range of area 614 (reverse horizontal position).

[0042] 6(b) and (c) show the positional relationship between the infrared light emitting diodes 166 and the eye 161 when the user places his / her eye close to the eyepiece 16. In this embodiment, the infrared light emitting diodes 166 are arranged at four locations on the inside of the eyepiece 16, as shown by diodes 166a to 166d in FIG. 6(b). As described above, the positional relationship between the gaze detection block 160 and the eye 161, i.e., the relative positional relationship between the infrared light emitting diodes 166 and the eye 161, affects the acquired gaze data.

[0043] In the case of digital camera 100, the user places his / her eye on eyepiece 16 while standing upright, so when the attitude of digital camera 100 changes, the relative positional relationship between infrared light emitting diode 166 (gaze detection block 160) and eye 161 also changes. In other words, it can be said that the attitude (direction) of digital camera 100 indicates the relative positional relationship between gaze detection block 160 and eye 161.

[0044] 6(b) shows the positional relationship between infrared emitting diodes 166 (diodes 166a to 166d) and eye 161 when digital camera 100 is in the landscape position. Diodes 166a and 166b emit infrared light from the upper eyelid side of eye 161, and diodes 166c and 166d emit infrared light from the lower eyelid side of eye 161.

[0045] 6(c) shows the positional relationship between the infrared light emitting diodes 166 (diodes 166a to 166d) and the eye 161 when the digital camera 100 is in the left vertical position. The diodes 166b and 166d irradiate the eye 161 from the upper eyelid side, and the diodes 166a and 166c irradiate the eye 161 from the lower eyelid side.

[0046] When digital camera 100 is in a right-hand vertical position (not shown), diodes 166a and 166c irradiate from the upper eyelid side of eye 161, and diodes 166b and 166d irradiate from the lower eyelid side of eye 161. When digital camera 100 is in a horizontally inverted position (not shown), diodes 166c and 166d irradiate from the upper eyelid side of eye 161, and diodes 166a and 166b irradiate from the lower eyelid side of eye 161.

[0047] In FIG. 6(a), the areas 611 to 614 are obtained by dividing 360° into four equal parts, and each range is set to 90°, but the range of each area may be set to a different size.

[0048] In general, calibration involves displaying multiple gaze points at different positions, acquiring and accumulating gaze data when the user gazes at each gaze point, and calculating a calibration value from the multiple gaze data. Since the multiple gaze points displayed at this time are displayed at different positions, gaze data of the eyeballs at various angles can be acquired. By registering the calculated calibration value as calibration data, the user can perform gaze input with higher accuracy without having to perform calibration every time he or she uses gaze input. In this embodiment, the calibration value calculated from the acquired gaze data is linked to the attitude (attitude information) of the digital camera 100 and saved / registered as calibration data.

[0049] The number of gaze points displayed in the calibration mode is known to be, for example, three or five. The display form of these gaze points includes, for example, the control described below. Among the multiple gaze points, one is displayed in sequence, and when gaze data at the first gaze point is acquired, the first point is hidden and the second point is displayed, and when all gaze data has been acquired and the calibration value has been calculated, the calibration is completed.

[0050] When performing calibration in the calibration mode, it is essential that the user gazes at the gaze point, and it is desirable to avoid an unstable gaze. For example, if only the first gaze point is displayed and the second point is displayed while the first point remains displayed, the user may unconsciously move their gaze back and forth between the first and second gaze points, resulting in an unstable gaze. Therefore, it is more desirable to display multiple gaze points one by one in sequence and obtain the corresponding gaze data.

[0051] In the calibration mode of this embodiment, gaze data is acquired at a total of five gaze points, and the gaze points are displayed one by one in sequence. Once all the gaze data has been acquired, they are hidden and the next gaze point is displayed.

[0052] Eyepiece detection unit 57 is an eyepiece detection sensor that detects (approach detection) the approach (eyepiece) and removal (eye separation) of eye (object) 161 to eyepiece unit 16 of the finder. System control unit 50 switches display unit 28 and EVF 29 between display (display state) and non-display (non-display state) depending on the state detected by eyepiece detection unit 57. More specifically, when at least digital camera 100 is in a shooting standby state and the switching setting of the display destination of the live view image captured by imaging unit 22 is set to automatic switching, when the eye is not placed near the camera, the display is turned on with display unit 28 as the display destination and EVF 29 is not displayed. Also, when the eye is placed near the camera, the display is turned on with EVF 29 as the display destination and display unit 28 is not displayed.

[0053] Eyepiece detection unit 57 can be, for example, an infrared proximity sensor, and can detect the approach of some object to eyepiece unit 16 of the finder that incorporates EVF 29. When an object approaches, infrared light projected from a light-projecting unit (not shown) of eyepiece detection unit 57 is reflected and received by a light-receiving unit (not shown) of the infrared proximity sensor. The amount of infrared light received can also be used to determine how close the object is to eyepiece unit 16 (eyepiece distance). In this way, eyepiece detection unit 57 performs eyepiece detection, which detects the proximity of an object to eyepiece unit 16.

[0054] In this embodiment, the light-projecting section and the light-receiving section of eye proximity detection unit 57 are separate devices from the aforementioned infrared light-emitting diode 166 and line-of-sight detection sensor 164. However, the infrared light-emitting diode 166 may also serve as the light-projecting section of eye proximity detection unit 57. Also, the line-of-sight detection sensor 164 may also serve as the light-receiving section.

[0055] The eye contact detection unit 57 detects that an object has been placed in contact with the eye when an object approaching within a predetermined distance of the eye piece 16 is detected from a non-eye contact state (non-approaching state). The eye contact detection unit 57 detects that an object that was detected as approaching has moved away from the eye piece state (approaching state) by a predetermined distance or more. The threshold for detecting the eye contact and the threshold for detecting the eye separation may be different, for example, by providing a hysteresis. In addition, after detecting the eye contact, the eye contact state is maintained until the eye separation is detected. After detecting the eye separation, the eye contact state is maintained until the eye contact is detected. Note that the infrared proximity sensor is only an example, and other sensors may be used for the eye contact detection unit 57 as long as they can detect the approach of an eye or an object that can be considered as an eye contact.

[0056] The system control unit 50 can detect the following operations or states based on the output from the line of sight detection block 160. The gaze of the user who has placed his / her eye close to the eyepiece unit 16 has been newly input (detected). In other words, gaze input has started. The user is receiving gaze input by placing his / her eye on the eyepiece unit 16. The user is holding the eyepiece 16 close to the object and gazing at it. The user who placed his / her eye close to the eyepiece unit 16 moves his / her gaze away from the eye. In other words, the gaze input ends. A state in which the user has placed their eye in the eyepiece 16 and is not inputting any gaze.

[0057] The term "gazing" as used herein refers to a case where the gaze position corresponding to the user's line of sight does not exceed a predetermined amount of movement within a predetermined period of time (below a predetermined threshold).

[0058] In this embodiment, a process of executing calibration based on a user's line of sight in a digital camera 100, which is one type of electronic device, will be described.

[0059] 3 is a control flowchart for when the user performs gaze calibration. This control process is realized by the system control unit 50 expanding a program stored in the non-volatile memory 56 into the system memory 52 and executing it.

[0060] The flowchart in FIG. 3 is started when the digital camera 100 is started and the user performs an operation to start calibration on the setting menu screen while looking through the finder, that is, while the user has placed his / her eye on the eyepiece 16. FIG. 7 shows an example of the display of the setting menu screen related to the gaze input. The setting of the gaze input shown in item 701 in FIG. 7 shows a state in which option 701a is selected, that is, the gaze input is set to ON. When an execution instruction is given for item 702, that is, when display 702a is selected and confirmed, the flowchart in FIG. 3 is started. In FIG. 7, display 702a displays "unregistered", which indicates that the user has never performed calibration. If the user has performed calibration even once, display 702a displays "registered". Regardless of which display is shown in display 702a, it is possible to perform calibration as described above.

[0061] In S301, the system control unit 50 displays a calibration start screen on the EVF 29. FIG. 5(a) shows an example of the display of the calibration start screen. FIG. 5(a) is an example of a display displayed on the EVF 29 when the user places his / her eye on the eyepiece unit 16 and the orientation of the digital camera 100 detected by the orientation detection unit 55 is in the landscape position (normal position). When detecting the user's line of sight including calibration, the user can make some adjustments to make it easier for the gaze detection block 160 to detect the user's line of sight. For this reason, a display like the dialogue 501 in FIG. 5(a) is displayed at the start of calibration. In addition, even for a user who has never used the gaze input function, displaying the dialogue 501 makes it easier to grasp the points of not only calibration but also gaze input. Note that the wording of the dialogue 501 is not limited to this, and it is also possible to skip S301 and start from S302 in response to an instruction to start calibration without displaying FIG. 5(a).

[0062] In S302, the system control unit 50 determines whether or not a confirmation operation has been performed. If a confirmation operation has been performed, the process proceeds to S303, and if not, the process proceeds to S304. In this embodiment, as shown in display 511 in Fig. 5(a), the confirmation operation is the pressing of the M-Fn button 77. Note that the confirmation operation is one of the conditions for the user to indicate his / her intention to proceed to the next step (screen), and is not limited to the pressing of the M-Fn button 77.

[0063] In S303, the system control unit 50 starts a process of acquiring calibration data. The process of acquiring calibration data will be described later with reference to the control flowchart of FIG.

[0064] In S304, the system control unit 50 displays a completion message that is displayed when the calibration is completed, instead of displaying the gaze point. Specifically, as shown in FIG. 5(g), the system control unit 50 displays a dialogue 503, options 512, and options 513.

[0065] In S305, the system control unit 50 determines whether or not there has been an instruction to perform calibration again. If there has been an instruction, the process returns to S400, and if not, the control flowchart in FIG. 4 ends. An instruction to perform calibration again refers to the selection of option 513 shown in FIG. 5(g). When option 513 is selected, calibration is performed from the beginning. On the other hand, when option 512 is selected, the calibration display screen transitions to the display screen before the user started calibration, for example, a setting menu screen.

[0066] By performing calibration multiple times, it is possible to obtain more accurate calibration data. By displaying the screen of FIG. 5(g), it is possible to inform the user that more accurate calibration data can be obtained by performing calibration multiple times, and it is possible to encourage the user to perform calibration multiple times. Note that, when the acquisition of gaze data at the five gaze points is completed, the calibration data acquisition process may be terminated without displaying the screen of FIG. 5(g). In other words, S304 and S305 may be skipped and the control flowchart of FIG. 3 may be terminated from S303.

[0067] In S306, the system control unit 50 judges whether or not an operation other than the confirmation operation has been performed. If an operation other than the confirmation operation, that is, an operation on an operating member other than the M-Fn button 77 described in S302 has been performed, the calibration process is terminated, and if not, the process returns to S302. If an operation on an operating member other than the M-Fn button 77 has been performed, a process corresponding to the operated member is performed. For example, when the playback button 79 is pressed, the process transitions to a playback mode process, and an image that has been captured or that has been stored in the recording medium 200 is played back. When the shutter button 61 is pressed, the process transitions to a shooting standby state, which is a shooting mode process, and shooting preparation processes such as AF, AE, and AWB are performed, and when a shooting instruction is given, a series of shooting processes are performed until the captured image is recorded in the recording medium 200 as an image file.

[0068] Fig. 4 is a control flowchart of the calibration data acquisition process described in S303 of Fig. 3. This process is started when the result of the determination in S302 of Fig. 3 is Yes.

[0069] In S400, the system control unit 50 initializes flag information F (F=0) and stores it in the system memory 52. ​​The flag information F is information indicating whether or not a change in posture is permitted during execution of calibration. When the flag information F is 0, the system control unit 50 notifies the user when a change in posture occurs during execution of calibration. Details will be described later.

[0070] In S401, the system control unit 50 sets a variable X to 1 (X=1) and stores it in the system memory 52. ​​In this embodiment, the variable X represents the number of gaze points to be displayed on the EVF 29.

[0071] In S402, the system control unit 50 refers to the non-volatile memory 56 and displays the Xth fixation point, which is a calibration display, on the EVF 29. The non-volatile memory 56 stores a plurality of calibration images that are associated in advance with the attitude of the digital camera 100 and the display order (X). When the digital camera 100 is in the landscape position (normal position), FIG. 5(b) is displayed on the EVF 29 when X=1, FIG. 5(c) is displayed on the EVF 29 when X=2, FIG. 5(d) is displayed on the EVF 29 when X=3, FIG. 5(e) is displayed on the EVF 29 when X=4, and FIG. 5(f) is displayed on the EVF 29 when X=5. That is, the display order of the fixation points is center, left, right, top, and bottom with respect to the user. The reason for the display order is to take into consideration the stability of the user's line of sight. As described above, instead of displaying the fixation points one by one in order as shown in FIG. 5, all five points may be displayed on the EVF 29 in this step and each point may be hidden in order. In other words, the gaze points may be hidden in order starting from the point where it is assumed that the user's gaze has ended (gaze data has been acquired), and the control flowchart of Figure 4 may be terminated when the gaze data for the last point has been acquired.

[0072] For example, if the gaze point is displayed below the EVF 29 and then above it, the user must move their gaze from bottom to top, which may impair the stability of the gaze due to the characteristics of the human eye. For this reason, when the eyepiece 16 is on the left and the grip 90 is on the top (left vertical position), the relative positions to the EVF 29 are center, left, right, top, and bottom, so the order is Fig. 5(b)→(e)→(f)→(d)→(c). When the eyepiece 16 is on the right and the grip 90 is on the bottom (right vertical position), the order is Fig. 5(b)→(f)→(e)→(c)→(d).

[0073] That is, regardless of the orientation of the digital camera 100, the display positions of the gaze points are displayed in the relative order of center, left, right, top, bottom as seen by the user. In this embodiment, the display order of the gaze points is as described above, and is a display form that changes depending on the orientation of the digital camera 100, but this is not limited to this. The number of gaze points may also be other combinations, such as four points on the top, bottom, left, and right, or three points on the top, left, bottom, and right.

[0074] In S403, similarly to S302, the system control unit 50 determines whether or not a confirmation operation has been performed. If a confirmation operation has been performed, the process proceeds to S404, and if not, the process proceeds to S426.

[0075] In S404, the system control unit 50 determines the orientation information P(X), which is the orientation of the digital camera 100 at the time when the confirmation operation was performed in S403. As described above, the orientation of the digital camera 100 is constantly detected by the orientation detection unit 55, but the orientation at the time when the confirmation operation, which is an expression of the user's intention, is performed is set as the orientation of the digital camera 100.

[0076] In S405, the system control unit 50 determines whether or not the variable X is 1. If X is 1, the process proceeds to S406, and if X is other than 1, the process proceeds to S408.

[0077] In S406, the system control unit 50 determines whether or not the attitude information P(X) of the digital camera 100 determined in S404 is an attitude that can be registered as calibration data. If it is an attitude that can be registered, the process proceeds to S407, and if not, the process proceeds to S416.

[0078] 6(a), the orientation that can be registered as calibration data is either the upright horizontal position, the left-portrait position, or the right-portrait position, and when the orientation of digital camera 100 is the inverted horizontal position, it is not possible to register the orientation as calibration data. This is because it is considered that there are few opportunities to take a picture using eye gaze input with eyepiece 16 within the range of area 614, but even in this inverted horizontal position, it may be possible to register the orientation as calibration data.

[0079] In S407, the system control unit 50 stores the orientation of the digital camera 100 confirmed in S404 as orientation information P(x) in the non-volatile memory 56. In this embodiment, only the orientation when the confirmation operation is performed while the first gaze point is displayed is stored in the non-volatile memory 56.

[0080] If the result of S405 is No, in S408, the system control unit 50 determines whether or not the orientation information P(X) of the digital camera 100 is the orientation P(1) when a confirmation operation is performed with the first gaze point displayed. If P(X) is P(1), the process proceeds to S409, and if not, the process proceeds to S421.

[0081] In S409, the system control unit 50 determines the gaze data E(x) when the Xth gaze point is displayed. In this embodiment, it is assumed that the gaze detection block 160 is used to constantly detect the gaze of the user during calibration. In S409, the system control unit 50 determines the gaze data E(X) as gaze information acquired at the timing when the confirmation operation is input in S403. The reason for determining the gaze data at the timing when the confirmation operation is performed is to obtain more reliable gaze data. Note that, even if there is no confirmation operation, the gaze data may be acquired at the timing when the gaze detection block 160 determines that there was gaze. When using gaze for the confirmation operation when data is acquired for calibration as in the control flow of FIG. 4, it is preferable to set a condition different from the condition for determining gaze when data is not acquired. For example, when data is not acquired, if the amount of movement of the user's gaze is equal to or less than a preset threshold within 120 milliseconds, it is determined that there was gaze. In contrast, when data is acquired, the amount of movement of the user's gaze is equal to or less than a regulated threshold within 3 seconds. Also, the gaze determination criterion time may remain at 120 milliseconds and the default threshold for the amount of gaze movement may be made smaller, or the time may be increased and the default threshold for the amount of gaze movement may be made smaller.

[0082] In S410, the system control unit 50 judges whether or not the gaze data E(X) confirmed in S409 can be used as gaze data (OK or NG). If it is OK, proceed to S411, and if it is NG, proceed to S416. The gaze data is NG when, when the confirmation operation is performed in S403, external light is reflected in the eyes or the lenses of glasses are reflected. In addition, there is a case where the appropriate gaze data E(X) for calculating a calibration value described later in S413 cannot be obtained due to reasons such as the user blinking frequently.

[0083] In S 411 , the system control unit 50 inputs X+1 to the variable X and stores it in the system memory 52 .

[0084] In S412, the system control unit 50 refers to the system memory 52 and judges whether or not X>Xth. If X>Xth, the process proceeds to S413, and if X≦Xth, the process proceeds to S414. Xth is a threshold value indicating the number of gaze points for which gaze data is obtained in the calibration. In this embodiment, Xth is set to 5 because the number of gaze points displayed is 5.

[0085] In S413, the system controller 50 calculates a calibration value from all of the gaze data E(X) acquired up to this step, and proceeds to S415.

[0086] In S414, the system control unit 50 hides the gaze point displayed on the EVF 29, and returns to S402. As described above, in the calibration mode of this embodiment, the gaze points, which are the calibration display, are displayed one by one in sequence, and control is performed so that only one gaze point is displayed on the EVF 29, so that the gaze point is hidden in this step.

[0087] In S415, system control unit 50 combines P(1) stored in S407 with the calibration value calculated in S413 to prepare calibration data, associates these data with each other, and stores them in non-volatile memory 56. As described above, a change in the relative positional relationship between gaze detection block 160 and eye 161 changes the position of the infrared light irradiated from infrared emitting diode 166 and reflected by eye 161, and the infrared light detected by gaze detection block 160 also changes. Therefore, by storing the orientation of gaze detection block 160, i.e., the attitude of digital camera 100, in association with the user's gaze data, the user can specify a position with a more accurate gaze input when executing a function using gaze input.

[0088] If it is determined in S406 that the posture information P(1) cannot be registered, or if it is determined in S410 that the gaze data E(X) is NG, the system control unit 50 displays a warning on the EVF 29 in S416. If it is determined No in S406, the warning is displayed as "Please operate in landscape or portrait orientation." If it is determined No in S410, a warning is displayed according to the cause of the gaze data E(x) being NG. For example, there is a lot of blinking, external light is reflected in the eyes, ghosts are obscured by the glasses, etc. In such cases, messages such as "Please keep both eyes wide open," "Do not blink when pressing the M-Fn button," and "If your glasses are down, raise the position of your glasses and then bring your eyes closer to the viewfinder" are displayed. In addition, messages such as "Please bring your eyes closer to the viewfinder" and "If another person uses the camera or if you put on or take off your glasses, please perform calibration with a different registration number" are displayed. By displaying such warnings, it is possible to encourage the user to improve their posture and eye condition so that gaze data can be acquired. The system control unit 50 displays the warning display on the EVF 29 while keeping the gaze point displayed. The system control unit 50 may perform control so that the warning display is displayed after temporarily hiding the gaze point.

[0089] In S417, the system control unit 50 determines whether or not a confirmation operation has been performed, similarly to S403. If a confirmation operation has been performed, the process proceeds to S418, and if not, the process proceeds to S419.

[0090] In S418, the system control unit 50 hides the warning displayed in S416, and returns to S402. Note that the timing for hiding the warning display (dialog 504 in FIG. 5(h)) displayed in S416 may be determined not based on the presence or absence of the confirmation operation in S417, but based on the lapse of a certain time from the start of display.

[0091] In S419, the system control unit 50 determines whether or not an operation other than the confirmation operation has been performed, similarly to S304. If an operation other than the confirmation operation has been performed, the system control unit 50 proceeds to S420, and if not, the system control unit 50 returns to S417.

[0092] In S420, the system control unit 50 discards the data acquired in the control flow chart of FIG. 4, i.e., the attitude P(1) of the digital camera 100 and the gaze data E(X) stored in the system memory 52, and ends the control flow charts of FIG. 4 and FIG. 3. In S420, since an instruction to end the data acquisition process was given before the acquisition of gaze data at all five points of the gaze point was completed, all the acquired data is discarded, and the control flow of FIG. 4 is ended. In other words, since an instruction to end the data acquisition process was given before the series of processes of calculating the calibration value from the acquired gaze data at all five points and storing the calibration value by linking it with the camera attitude information was completed, all the acquired data is discarded. By controlling in this way, it is possible to prevent the calibration value from being calculated even though the gaze data, which is gaze information, is insufficient as the number of data for calculating the calibration value. This makes it possible to avoid the acquisition and storage of a calibration value with low accuracy, i.e., calibration data.

[0093] If it is determined in S408 that the Xth point posture information P(X) is different from the first point posture information P(1) (No), the system control unit 50 determines in S421 whether or not flag information F is 1. If flag information F is 1, the process proceeds to S409. If flag information F is not 1 (flag information F is 0), the process proceeds to S422.

[0094] If it is determined in S408 that the X-th point posture information P(X) is different from the first point posture information P(1) (No), the system control unit 50 displays a warning in S422. FIG. 5(h) shows an example of the display. By displaying the warning, the system control unit 50 notifies the user that the current posture is different from the posture when the first point of gaze data was acquired. In addition, by displaying a warning such as the dialogue 504 shown in FIG. 5(h) as the warning display, it is possible to prompt the user to return the posture to the posture when the first point of gaze data was acquired. In this embodiment, as shown in FIG. 5(h), the warning display is performed while continuing to display the gaze point. By controlling in this way, when the dialogue 504 is hidden, the user does not need to move his / her gaze to find the gaze point. However, control may be performed so that both the gaze point and the warning are not displayed on the EVF 29.

[0095] In S423, the system control unit 50 determines whether or not the attitude of the digital camera 100 has returned to normal. The system control unit 50 monitors the output data of the attitude detection unit 55 and determines whether or not the same attitude information as P(1) has been acquired. If the same attitude information as P(1) has been acquired, the process proceeds to S426; if not, the process proceeds to S424.

[0096] In S424, similarly to S403, the system control unit 50 judges whether or not a confirmation operation has been performed. If a confirmation operation has been performed, that is, if the M-Fn button 77 shown in the display 514 of Fig. 5(h) has been pressed, the process proceeds to S425, and if not, the process returns to S423.

[0097] In S425, the system controller 50 sets the flag information F to 1.

[0098] From the determination of Yes in S424, it can be assumed that the user wants to ignore the warning display and continue the calibration in a different orientation from P(1). Therefore, in the subsequent processing, even if the orientation information P(X) acquired in S403 is different from the orientation information P(1) of the first point, the warning display is not repeated and acquisition of the gaze data E(X) can be continued. For example, P(1) is in the normal position, and P(2) and P(3) are also in the normal position, but P(4) changes to the left vertical position, and a confirmation operation is performed in S424 while the position remains in the left vertical position. In this case, the system control unit 50 hides the warning and returns to S402 to continue displaying the fourth gaze point (X=4). Then, even if the orientation of the housing of the digital camera 100 remains in the left vertical position and a confirmation operation is performed in S403, the gaze data E(X) is accumulated as gaze data corresponding to the Xth gaze point by proceeding to S409 in S4XX.

[0099] By such control, the accuracy of the acquired and accumulated gaze data decreases with respect to the attitude of the digital camera 100, but the annoyance of returning to the first gaze point even though calibration has been performed up to the fourth point can be reduced. If the gaze point displayed after the attitude of the camera changes returns to the first point (the gaze point displayed first), the user is likely to feel annoyed at starting the calibration from the beginning. This may result in the gaze input function not being used at all or the gaze input being used without performing calibration. If the gaze input is used without performing calibration, a difference occurs between the gaze of the user and the gaze position corresponding to the gaze detected by the digital camera 100, which is inconvenient to use, and the user may end up not using the gaze input. Therefore, it is possible to promote the use of the gaze input function while promoting the execution of calibration to allow for more accurate gaze input without feeling annoyed.

[0100] Note that the determination process of S423 may be repeated until the attitude of digital camera 100 becomes equivalent to P(1) even if a confirmation operation is performed. In this case, since it is possible to collect gaze data when the attitude of the body of digital camera 100 is approximately the same, it is possible to acquire and register calibration data with higher accuracy. Therefore, it is possible to reduce the error between the position where the user is looking and the gaze position corresponding to the gaze calculated by calibration.

[0101] In S426, the system controller 50 hides the warning displayed in S422, and returns to S402.

[0102] After the image showing the Xth fixation point is displayed, if no confirmation operation is performed and S403 returns No, the system control unit 50 determines in S427 whether or not an operation other than the confirmation operation has been performed, similar to S304. If an operation other than the confirmation operation has been performed, the control flow charts of Figs. 4 and 3 are terminated, and if not, the process returns to S403.

[0103] In this embodiment, the confirmation operation is the pressing of the M-Fn button 77, but as described above, this is one of the conditions for the user to indicate his / her intention to proceed to the next step (screen), and is not limited to the pressing of the M-Fn button 77. Any other operation member having a function to which the confirmation function can be assigned may be used. In the case of using another operation member, it is easier to operate if the operation member is located in a position where the user can operate the operation member while placing his / her eye on the eyepiece 16 and holding the grip part 90 with his / her right hand. For example, even if the operation member is located on the lens barrel, which is the exterior of the lens unit 150, the user can press the operation member with his / her left hand while placing his / her eye on the eyepiece 16 and holding the grip part 90 with his / her right hand. The confirmation operation may be performed not by pressing the operation member, but by the user gazing at a displayed gaze point. In other words, it is also possible to determine that a confirmation operation has been performed by the user when, as a condition other than the pressing of the M-Fn button 77, the user's gaze is at the gaze point displayed on the EVF 29 and does not exceed a predetermined amount of movement within a predetermined time (below a predetermined threshold value). In the case of such control, the gaze data is not confirmed until the user's gaze stabilizes (until it is determined that the gaze data is stable enough to be acquired), so there is a low possibility that the gaze data (E(X)) will be determined to be NG in FIG. 4, S410 in this embodiment.

[0104] Furthermore, the warning displayed in S416 and S422 may be an icon instead of a text, and may be something that can notify the user, such as a change in the display form or color of the EVF 29, blinking of the EVF 29, or a combination of these. For example, the dialogue 504 in Fig. 5(h) may be made to blink, or the background color of the EVF 29 may be made different from the background color when the gaze point is displayed.

[0105] In this manner, in this embodiment, if the posture of the housing changes during calibration, a warning is displayed, the user is prompted to return to the posture when the first gaze data was confirmed, and gaze data after the posture change is not accepted until the user gives an instruction. This allows the user to reduce the decrease in accuracy of the gaze data, and allows more accurate calibration data to be created and acquired. Even if the posture has changed, if the user performs a confirmation operation, the gaze point displayed before the posture change continues to be displayed without returning to the first gaze point, thereby reducing the possibility that the user will find the execution of calibration annoying. Therefore, it is possible to acquire data without the user feeling that the execution of calibration is annoying, while reducing the decrease in accuracy of the calibration data.

[0106] The various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0107] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. The shape of the gaze point when performing calibration is a square shape, but any shape that the user can gaze at may be used. The display may not be a static display, but may be an animation-like display. In addition, the background on which the gaze point is displayed is a single color, but the color of the background may change, a gradation display may be used, or a playback image or an LV image may be displayed. For example, the color or brightness of the background may be changed each time multiple gaze points are displayed so that the user does not have to perform calibration multiple times in bright and dark places, and calibration may be performed under various conditions in one calibration.

[0108] In this embodiment, the case where the present invention is applied to a digital camera 100 having an EVF 29 has been described as an example, but this is not limited to this example, and the present invention can be applied to any electronic device having an acquisition means capable of acquiring information about the user's line of sight. Specifically, the present invention can be applied to any electronic device having an acquisition means capable of acquiring information about the user's line of sight by displaying a gaze point for performing calibration on the display unit and detecting the line of sight of the user looking at the display unit. In addition, each embodiment can be appropriately combined. In this embodiment, the eyepiece unit 16 having the EVF 29 and the line of sight detection block 160 are integrated into one unit, but this embodiment can also be implemented with a configuration in which the display unit and the line of sight detection unit are separate devices and are connected to detect the user's line of sight. That is, the present invention can be applied to personal computers, PDAs, mobile phone terminals and tablets including smartphones, portable image viewers, printer devices equipped with displays, digital photo frames, music players, and wearable devices such as game consoles, electronic book readers, and head-mounted displays.

[0109] For example, when an external gaze detector is connected to a personal computer, the present embodiment is controlled based on the orientation of the gaze detector, not the orientation of the personal computer. Consider a case where calibration is started with the gaze detector placed horizontally and user gaze data is acquired. If the orientation of the gaze detector is changed from horizontal to vertical during calibration, control is performed so that the user gaze data after the gaze detector orientation becomes vertical is not acquired or saved.

[0110] The present invention is also applicable not only to the imaging device itself, but also to a control device that communicates with an imaging device (including a network camera) via wired or wireless communication and remotely controls the imaging device. Examples of devices that remotely control an imaging device include a smartphone, a tablet PC, and a desktop PC. The imaging device can be remotely controlled by notifying the control device of commands that cause the imaging device to perform various operations and settings based on operations or processes performed on the control device. Also, a live view image captured by the imaging device may be received via wired or wireless communication and displayed on the control device.

[0111] The various controls described above as being performed by the system control unit 50 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0112] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (program) that realizes the functions of the above-mentioned embodiments is supplied to a system or device via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the recording medium on which the program is stored constitute the present invention.

Claims

[Claim 1] A detection means for detecting a user's line of sight with respect to a screen; a control means for controlling execution of calibration based on information regarding the line of sight of the user detected by the detection means, The electronic device according to claim 1, wherein the control means performs control so as to give a notification regarding the calibration when a predetermined condition regarding the calibration is satisfied.

Images