Imaging device, control method thereof, and storage medium

Abstract

An imaging device includes a gaze detection unit configured to detect a user's gaze input; a calibration unit configured to perform calibration on the gaze input and set gaze input calibration data; an orientation detection unit configured to detect an orientation of the imaging device; and a control unit configured to perform control such that, when the orientation detection unit detects that the imaging device is in a first orientation, and the gaze input calibration data is not set, a first display is displayed on a display to prompt to perform calibration on the gaze input, and when the orientation detection unit detects that the imaging device is in a second orientation different from the first orientation, the first display is not displayed.

Description

Technical Field

[0001] The present invention relates to an electronic device, its control method, program and storage medium, which can be used in various orientations of its housing and can control the display according to the orientation and settings of the electronic device. Background Technology

[0002] A digital camera is known to be configured to detect the gaze of a user taking an image with the camera and to detect the position (area) in the viewfinder that the user is viewing, thereby controlling imaging functions such as autofocus adjustment. Patent Document 1 discloses that, when setting calibration data based on the direction of the user's gaze to match the user's eye rotation angle with the point of fixation, the calibration data is set to be correlated with the posture (orientation) of the optical device.

[0003] Citation List Patent documents Patent Document 1: Japanese Patent Application Publication No. 07-255676 Summary of the Invention

[0004] Technical issues However, in the technology disclosed in Patent Document 1, when user gaze input is used in a digital camera, the user cannot recognize the fact that if calibration data is not set, a discrepancy exists between the user's current viewing position and the detected gaze position (leading to reduced gaze detection accuracy). Even when a display is provided to prompt the user for settings, depending on the orientation of the digital camera, the display may be annoying to the user. The user may not realize that, regardless of whether gaze input is used in a digital camera or whether an image is being captured, they may expect to make settings to control the functions of a device such as a smartphone. Depending on the device's orientation, the display prompting the user for such settings can be unpleasant.

[0005] In view of the above, the object of the present invention is to provide a display that prompts the user to set and control functions related to electronic devices at appropriate time intervals.

[0006] Solution to the problem To address the aforementioned problems, the present invention provides a camera device, comprising: a gaze detection unit configured to detect a user's gaze input; a calibration unit configured to perform calibration on the gaze input and set gaze input calibration data; an orientation detection unit configured to detect the orientation of the camera device; and a control unit configured to perform control such that, when the orientation detection unit detects that the camera device is in a first orientation and no gaze input calibration data is set, a first display is shown on a screen to prompt for gaze input calibration, and when the orientation detection unit detects that the camera device is in a second orientation different from the first orientation, the first display is not shown.

[0007] Beneficial effects of the invention According to the present invention, a display can be provided at appropriate time intervals to prompt the user to set functions related to controlling electronic devices. Attached Figure Description

[0008] [ Figure 1A The image shows an external view of a camera according to an embodiment.

[0009] [ Figure 1B The image shows an external view of a camera according to an embodiment.

[0010] [ Figure 2 [Illustration 1] is a functional block diagram illustrating the construction according to an embodiment.

[0011] [ Figure 3 [ ] is a flowchart illustrating the control processing of a digital camera.

[0012] [ Figure 4 [ ] is a flowchart illustrating the control processing in the calibration mode processing of a digital camera.

[0013] [ Figure 5 [This is a flowchart illustrating the control process of the orientation and display guidance of the digital camera 100 in image capturing mode processing.]

[0014] [ Figure 6A The image shows a menu screen for setting functions related to line of sight, according to an embodiment.

[0015] [ Figure 6B The image shows a menu screen for setting functions related to line of sight, according to an embodiment.

[0016] [ Figure 7A [Illustration 1] is a diagram illustrating an example of a calibration mode process according to an embodiment.

[0017] [ Figure 7B [Illustration 1] is a diagram illustrating an example of a calibration mode process according to an embodiment.

[0018] [ Figure 7C [Illustration 1] is a diagram illustrating an example of a calibration mode process according to an embodiment.

[0019] [ Figure 7D [Illustration 1] is a diagram illustrating an example of a calibration mode process according to an embodiment.

[0020] [ Figure 7E [Illustration 1] is a diagram illustrating an example of a calibration mode process according to an embodiment.

[0021] [ Figure 7F [Illustration 1] is a diagram illustrating an example of a calibration mode process according to an embodiment.

[0022] [ Figure 7G [Illustration 1] is a diagram illustrating an example of a calibration mode process according to an embodiment.

[0023] [ Figure 8A [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0024] [ Figure 8B [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0025] [ Figure 8C [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0026] [ Figure 8D [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0027] [ Figure 8E [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0028] [ Figure 8F [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0029] [ Figure 8G [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0030] [ Figure 8H [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0031] [ Figure 8I [This is a diagram showing an example of the orientation display and guidance display of the digital camera 100.]

[0032] [ Figure 9A[Illustration 1] is a diagram illustrating a variant of applying display control to a smartphone according to an embodiment.

[0033] [ Figure 9B [Illustration 1] is a diagram illustrating a variation of applying display control to a smartphone according to an embodiment.

[0034] [ Figure 9C [Illustration 1] is a diagram illustrating a variation of applying display control to a smartphone according to an embodiment.

[0035] [ Figure 9D [Illustration 1] is a diagram illustrating a variant of applying display control to a smartphone according to an embodiment.

[0036] [ Figure 9E [Illustration 1] is a diagram illustrating a variation of applying display control to a smartphone according to an embodiment.

[0037] [ Figure 9F [Illustration 1] is a diagram illustrating a variation of applying display control to a smartphone according to an embodiment. Detailed Implementation

[0038] <Exterior View of a Digital Camera 100> Embodiments of the present invention will now be described with reference to the accompanying drawings.

[0039] Figure 1A and Figure 1B Each of the above shows an external view of a digital camera 100, which is an example of a device according to the present invention. Figure 1A A tilted front view of the digital camera 100 is shown, while Figure 1B A tilted rear view of the digital camera 100 is shown. Figure 1A and Figure 1B In this setup, display unit 28 is located on the back of the camera and is configured to display images and various information. Touch panel 70a is a touch-operable component that can detect touch operations on the display surface (operation surface) of display unit 28. External viewfinder display unit 43 is a display unit located outside the camera's viewfinder and is configured to display shutter speed, aperture value, and various other camera settings.

[0040] The shutter button 61 is an operating component used to issue an image capture command. The mode selection switch 60 is an operating component used to switch between various modes. The terminal cover 40 is a cover protecting the connector (not shown) of the connecting cable connecting the digital camera 100 and external devices. The main electronic dial 71 is a rotary operating component included in the operation unit 70. By rotating the main electronic dial 71, the shutter speed, aperture value, and other settings can be changed. The power switch 72 is an operating component used to turn the power of the digital camera 100 on / off. The secondary electronic dial 73 is a rotary operating component included in the operation unit 70 and is capable of moving the selection box, moving the image frame forward / backward, etc. The cross key 74 is included in the operation unit 70 and is an operating component with a press button that allows pressing in four directions. Different operations are performed depending on the direction the cross key 74 is pressed. The setting button 75 is a press button included in the operation unit 70 and is mainly used for selecting options. The movie button 76 is used to start and stop shooting (recording) moving images. An AF-ON button 77 is included in the operation unit 70 and is used to initiate AF operation. AF operation primarily begins when the shutter button 61 is pressed, but the AF-ON button also issues a command to initiate AF operation when pressed. In digital camera 100, the AF start command and image capture command can be separated when the shutter button 61 is pressed, provided that AF operation is not performed. By pressing the shutter button 61 after pressing the AE lock button 78, an image can be captured while the AF position is locked, and an image can also be captured even when AF operation is not possible. The AF-ON button 77 is included in the operation unit 70. When the AF-ON button 77 is pressed in image capture standby mode (where no preparation for image capture is being made and the camera is waiting to begin capturing an image using the image capture unit 22 in image capture mode), the exposure conditions are locked. That is, an image can be captured using the exposure value locked by the user. The playback button 79 is an operation button included in the operation unit 70 and is used to switch between image capture mode and playback mode. When the playback button 79 is pressed in image capture mode, the operation mode changes to playback mode. In playback mode, the latest image recorded on the recording medium 200 is displayed on the display unit 28. When the menu button 81 included in the operation unit 70 is pressed, a menu screen is displayed on the display unit 28, and various settings can be made. The multi-controller 65 is an operating component that includes a directional switch operable in eight directions and a pressable button. Various operations can be performed depending on the tilt direction of the multi-controller 65. The user can intuitively make various settings on the menu screen displayed on the display unit 28 using the cross key 74, the setting button 75, and the multi-controller 65.The gaze determination button 82 is a press-button included as an operating member in the operation unit 70, and is used to select or deselect a subject based on the position indication of the gaze pointer, as will be described later. The gaze determination button is positioned such that it can be operated by holding the grip 90 with the right thumb, allowing the user to easily operate the gaze determination button even when the user is looking through the viewfinder (when the user's eye is in contact with the eyepiece 16).

[0041] The operation unit 70 is a collection of various operating components used as an input unit for receiving operations from the user. The operation unit 70 includes push-buttons, rotary dials, touch sensors, etc. The operation unit 70 includes at least the following: shutter button 61, touch panel 70a, main electronic dial 71, power switch 72, secondary electronic dial 73, cross keys 74, setting button 75, animation button 76, AF-on button 77, AE lock button 78, playback button 79, menu button 81, line-of-sight button 82, multi-controller 65, filter button 84, and framing assist button. Regarding the line-of-sight function, which moves the AF frame to the position indicated by the line of sight and will be described below, a dedicated button for it can be installed on the digital camera 100, or the line-of-sight function can be assigned to an operating component that also has another function. Examples of operating components that can be assigned the line-of-sight function include: AF-on button 77, AE lock button 78, push-button in multi-controller 65, filter button 84, and framing assist button. The AF-on button 77, AE-lock button 78, and multi-controller 65 are positioned so that the user can operate them with his / her right index finger while operating the shutter button 61 without interfering with image capture. More specifically, when the user grips the hold 90 with his / her right hand, the aforementioned operating components are located above the center of the back of the digital camera 100 on the side opposite to the subject (i.e., the side where the shutter button is located), and to the right of the EVF 29 (on the side of the hold 90). Their positions are not limited to the back of the digital camera 100, as long as the user can operate them while operating the shutter button 61. For example, the filter button 84, the viewfinder assist button, or similar operating components can be located on the front (on the subject side), or some operating components can be disposed on the lens unit 150. In this case, the position of the operating components is not limited to the back of the digital camera 100 as described above, but the operating components can be located anywhere, as long as they can be operated by the index finger or other fingers of the user's right hand operating the shutter button 61. Examples of operating components with other functions (to which the user can assign a gaze-determining function) include: a button that does not cause a switch from image capture mode when the shutter button 61 is being operated, and a function button that does not interfere with the image capture function when the shutter button 61 is operated. The gaze-determining function can be assigned to a button that can be pressed and can be assigned various functions. Operating components for the above purposes are not limited to press buttons, but can include rotatable operating rings, left- or right-operable levers, etc. Alternatively, the touch panel 70a that detects pressure can be configured such that when the touch panel 70a is pressed with strong pressure, the above functions are performed.

[0042] Communication terminal 10 is a communication terminal for communication between digital camera 100 and lens unit 150 (removable) described below. Eyepiece 16 is the eyepiece of eyepiece viewfinder (view-in viewfinder). The user can view the image displayed on EVF (electronic viewfinder) 29 (used as an in-viewfinder display unit) through eyepiece 16. Eye contact detection unit 57 is an eye contact detection sensor that detects whether the user's eye is in contact with eyepiece 16 while taking an image. Lid 202 is a cover for the slot where recording medium 200 is placed. Grip 90 is a grip shaped to be easy to hold with the right hand when the user holds digital camera 100. Shutter button 61 and main electronic dial 71 are located in a position that allows them to be operated with the index finger of the right hand while holding the digital camera by gripping grip 90 with the little finger, ring finger, and middle finger of the right hand. In this state, secondary electronic dial 73 is located in a position that can be operated with the thumb of the right hand.

[0043] Figure 2 This is a block diagram illustrating an example of the construction of a digital camera 100 according to this embodiment. Figure 2 In this design, lens unit 150 is a lens unit that includes an interchangeable camera lens. Lens 103 typically includes multiple lenses, but for simplicity, it is shown here as including only one lens. Communication terminal 6 is a communication terminal for communication between lens unit 150 and digital camera 100. Lens unit 150 communicates with system control unit 50 via communication terminal 6 and the aforementioned communication terminal 10, and the internal lens system control circuit 4 controls aperture 1 via aperture drive circuit 2. Then, lens 103 moves via AF drive circuit 3 to adjust focus.

[0044] Shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

[0045] The camera unit 22 is a camera device that includes a CCD device, CMOS device, or the like configured to convert optical images into electrical signals. The A / D converter 23 is used to convert the analog signals output from the camera unit 22 into digital signals.

[0046] The image processing unit performs predetermined pixel interpolation, resizing (such as reduction processing), and color conversion processing on the data provided from the A / D converter 23 or the memory control unit 15 (described below). Furthermore, the image processing unit 24 performs predetermined calculations on the captured image data. The system control unit 50 performs exposure control and range control based on the results of the calculations performed by the image processing unit 24. More specifically, AF (autofocus), AE (auto exposure), and EF (flash pre-fire) processing are performed using the TTL (through-lens) method. The image processing unit 24 also performs predetermined calculations on the captured image data and performs AWB (automatic white balance) processing using the TTL method based on the obtained calculation results.

[0047] The memory control unit 15 controls the data transmission between the A / D converter 23, the image processing unit 24, and the memory 32. Output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data acquired by the camera unit 22 and converted into digital data by the A / D converter 23, as well as image data for display on the display unit 28 and the EVF 29. The memory 32 has sufficient storage capacity to store a predetermined number of still images or a predetermined length of moving images and sound.

[0048] The memory 32 also serves as a memory for displaying images (video memory). Display image data written to the memory 32 is displayed by the display unit 28 or EVF 29 via the memory control unit 15. The display unit 28 and EVF 29 display images on display devices such as LCDs or OLEDs based on signals from the memory control unit 15. Live View (LV) display can be performed by sequentially transmitting and displaying data that has been A / D converted by the A / D converter 23 and stored in the memory 32 to the display unit 28 or EVF 29. Hereinafter, the image displayed through live view processing will be referred to as a live view image (LV image).

[0049] An infrared LED 166 is a light-emitting element used to detect the user's gaze on the viewfinder screen and emits infrared light toward the user's eyeball (eye) 161. The infrared light emitted from the infrared LED 166 is reflected by the eyeball (eye) 161, and the reflected infrared light reaches a dichroic mirror 162. The dichroic mirror 162 reflects only infrared light and transmits visible light. The reflected infrared light, with its altered optical path, forms an image on the imaging surface of the gaze detection sensor 164 via a camera lens 163. The camera lens 163 is an optical component constituting the gaze detection optical system. The gaze detection sensor 164 includes an imaging device such as a CCD-type image sensor.

[0050] The gaze detection sensor 164 photoelectrically converts incident reflected infrared light into an electrical signal and outputs the resulting electrical signal to the gaze detection circuit 165. The gaze detection circuit 165 includes at least one processor and is configured to detect the user's gaze based on the output signal from the gaze detection sensor 164, according to the image or movement of the user's eyeball (eye) 161, and output the detection information to the system control unit 50. Therefore, the dichroic mirror 162, camera lens 163, gaze detection sensor 164, infrared LED 166, and gaze detection circuit 165 form a gaze detection block 160. The gaze detection block 160 is one of the receiving units that receives gaze input.

[0051] In this invention, a gaze detection block 160 uses a corneal reflection method to detect gaze. In the corneal reflection method, the movement of the user's eye and the direction of the user's gaze (gaze) are detected based on the positional relationship between the reflected infrared light emitted from the infrared light-emitting diode 166 and the pupil of the eyeball (eye) 161, and more specifically, based on the positional relationship between the reflected light on the cornea and the pupil of the eyeball (eye) 161. Various other methods exist for detecting gaze, such as a method called scleral reflection, which uses the difference in light reflectivity between the iris and the sclera. Other gaze detection methods can be used, as long as they are capable of detecting gaze.

[0052] Various camera settings, including shutter speed and aperture value, are displayed on the external display unit 43 via the external display unit drive circuit 44.

[0053] The non-volatile memory 56 is an electrically erasable and rewritable memory such as a flash ROM. The non-volatile memory 56 stores constants, programs, etc., for the operation of the system control unit 50. A program refers to a program used to execute the various flowcharts described below according to this embodiment.

[0054] The system control unit 50 includes circuitry or at least one processor and controls the entire digital camera 100. The various processes described below according to this embodiment are implemented by executing programs stored in the aforementioned non-volatile memory 56. For example, RAM is used as system memory 52, and constants, variables, and programs for the operation of the system control unit 50 are read from the non-volatile memory 56 and loaded into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, display unit 28, etc.

[0055] System Timer 53 is a time measurement unit that measures time in various controls and the time of the built-in clock.

[0056] The mode selection switch 60 is an operating component included in the operation unit 70, and switches the operation mode of the system control unit 50 between still image shooting modes, moving image shooting modes, etc. The still image shooting modes include automatic image shooting mode, automatic scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes available in image shooting settings for various scenarios. The mode selection switch 60 allows the user to directly switch to one of these modes. Alternatively, after using the mode selection switch 60 to switch to the list of image shooting modes once, the user can select one of the multiple modes displayed and switch to the selected mode using another operating component. Similarly, the moving image shooting mode can include multiple modes.

[0057] The first shutter switch 62 is activated during operation of the shutter button 61 on the digital camera 100. That is, the first shutter switch 62 is activated when the shutter button 61 is half-pressed (to issue a command to prepare to capture an image). When the first shutter switch 62 is activated, a first shutter switch signal SW1 is generated. The first shutter switch signal SW1 causes image capture preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (flash pre-emission) processing to begin.

[0058] When the shutter button 61 is fully pressed to the so-called fully pressed position (to issue an image capture command), the second shutter switch 64 is activated, and a second shutter switch signal SW2 is generated. In response to the second shutter switch signal SW2, the system control unit 50 begins a series of image capture processes, from reading signals from the imaging unit 22 to writing the captured image as an image file to the recording medium 200. While the second shutter switch remains activated, the digital camera 100 performs continuous image capture operations (continuous shooting) at a speed according to a predetermined permissible continuous image capture speed.

[0059] The power control unit 80 includes a battery detection circuit, a DC-DC converter, a switching circuit for switching which blocks are to be powered, and detects whether a battery is installed, the type of battery, and the remaining battery power. Based on the detection results and instructions from the system control unit 50, the power control unit 80 controls the DC-DC converter and provides the required voltage to each part, including the recording medium 200, for the required time period. The power supply unit 30 includes a primary battery such as an alkaline or lithium battery, a secondary battery such as a NiCd, NiMH, or Li battery, and an AC adapter.

[0060] Recording medium I / F 18 is an interface with recording medium 200, such as a memory card or hard disk. Recording medium 200 is a recording medium, such as a memory card, used to record captured images, and recording medium 200 can be implemented, for example, by a semiconductor memory or a hard disk.

[0061] The communication unit 54 transmits / receives video and / or audio signals via a wireless or wired cable connection. The communication unit 54 can also connect to a wireless LAN (Local Area Network) and the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images captured by the camera unit 22 (including live view images) and images recorded on the recording medium 200, and can receive images and various information from external devices.

[0062] The posture detection unit 55 detects the posture (orientation) of the digital camera 100 relative to the direction of gravity. Based on the posture detected by the posture detection unit 55, it can be determined whether the image captured by the imaging unit 22 is an image captured by the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the posture detected by the posture detection unit 55 to the image file of the image captured by the imaging unit 22, or it can rotate the image and record the resulting rotated image. An accelerometer, a gyroscope, or the like can be used as the posture detection unit 55. The movement of the digital camera 100 (e.g., panning, tilting, lifting, whether stationary, etc.) can also be detected by using an accelerometer or gyroscope sensor used as the posture detection unit 55.

[0063] The eye contact detection unit 57 is an eye contact sensor configured to detect the approach (eye contact) and departure (eye separation) of an eye (object) 161 toward / away from the viewfinder eyepiece 16. The system control unit 50 switches the display unit 28 and EVF 29 between a display on state and a display off state based on the state detected by the eye contact detection unit 57. More specifically, at least when the digital camera 100 is in standby mode for capturing images, and when the display destination switching setting for the live view image captured by the imaging unit 22 is in automatic switching mode, the display unit 28 is designated as the display destination, and the display unit 28 is turned on and the EVF 29 is turned off when the eye is not in contact with the eyepiece. When the eye comes into contact with the eyepiece, the EVF 29 is set as the display destination device, and the EVF 29 is turned on and the display unit 28 is turned off. For example, an infrared proximity sensor can be used as the eye contact detection unit 57 to detect any object approaching the viewfinder eyepiece 16, which includes the EVF 29. When an object approaches, infrared light projected from the projection portion (not shown) of the eye contact detection unit 57 is reflected by the object, and the reflected infrared light is received by the light receiving portion (not shown) of the infrared proximity sensor. Based on the amount of infrared light received, it can be determined how close the object is to the eyepiece 16 (i.e., eye distance can be detected). As described above, the eye contact detection unit 57 performs eye contact detection processing to detect the distance of an object approaching the eyepiece 16. In this embodiment, it is assumed that the light projection portion and light receiving portion of the eye contact detection unit 57 are implemented by devices different from the infrared light-emitting diode 166 and the gaze detection sensor 164. However, the infrared light-emitting diode 166 can also be used as the light projection portion of the eye contact detection unit 57. The gaze detection sensor 164 can also be used as the light receiving portion. In a state where there is nothing on the eyepiece (i.e., nothing is close to the eyepiece), if an object approaching the eyepiece is detected within a predetermined distance range, it is determined that the eyepiece is in an eye contact state. When the object in the eye contact state (eye proximity state) moves away by an amount greater than or equal to the predetermined distance, it is determined that the object has left. The thresholds used to detect eye contact and eye separation can be different. This can be achieved, for example, by providing hysteresis. It is assumed that after eye contact is detected, the eye contact state is maintained until eye separation is detected. It is also assumed that after eye separation is detected, the eye separation state is maintained until eye contact is detected. The use of an infrared proximity sensor is merely an example, and other sensors can be used as the eye contact detection unit 57, as long as they can detect an eye or object approaching the eyepiece.

[0064] The system control unit 50 can detect the following operations or states based on the output from the line-of-sight detection block 160.

[0065] • The gaze detection block 160 detects the state of the user's gaze when the eyes are in contact with the eyepiece 16, that is, there is a gaze input state.

[0066] • The gaze detection block 160 detects the user's gaze when the eyes are in contact with the eyepiece 16 and the user is looking at a certain point (described later).

[0067] • After the gaze detection block 160 detects the user's gaze, the gaze disappears and is no longer detected, that is, the gaze input is complete.

[0068] • The gaze detection block 160 did not detect the state of the user's gaze when the eyes were in contact with the eyepiece 16, that is, it did not detect the state of input through the gaze.

[0069] The aforementioned state in which the user is looking refers to the following state: the gaze detection block 160 identifies the position (gaze position) on the EVF 29 based on the detected gaze, and determines that the identified gaze position has not changed beyond a predetermined movement range within a predetermined time.

[0070] The completion of gaze input can occur when the user's eyes are removed from the eyepiece 16, or when the user's eyelids are closed and therefore the user's eyes 161 cannot be detected while the eyes remain in contact with the eyepiece 16.

[0071] Calibration is a step that uses the gaze detection block 160 to detect the user's gaze and performs calibration to more accurately determine the location of the gaze directed by the user. Without calibration, the gaze detection block 160 can detect the user's gaze and determine the location of the gaze. However, the overall structure of the human eye, including the eyelids, varies from person to person. Therefore, depending on the user, it may be difficult to determine the location of the gaze. By performing calibration, gaze data, which is user-specific gaze information for the digital camera 100, can be obtained. By calculating calibration values ​​based on the obtained user-specific gaze data, the location of the gaze given by the user can be determined more accurately. Assuming gaze detection is performed in various orientations, such as in the case of the digital camera 100 according to this embodiment, there is a possibility that the positional relationship between the EVF 29 of the digital camera 100 and the eye 161 may change; that is, the relative positional relationship between the gaze detection block 160 and the eye 161 may change. To address this situation, it is desirable to perform calibration for both vertical and horizontal orientations of the digital camera 100. By calculating the gaze position using calibration values ​​given in each orientation, the difference between the position seen by the user and the detected gaze position can be minimized. In this embodiment, if the digital camera 100 has already been calibrated in the horizontal or vertical orientation (CAL data has been acquired), the acquired CAL data (described below) can be used to detect the gaze with fairly high accuracy.

[0072] During calibration, multiple gaze points are displayed at different locations, and gaze data is acquired as the user gazes at each gaze point. The gaze data is accumulated, and a calibration value is calculated based on the multiple gaze data points. Because multiple gaze points are displayed at different locations, gaze data can be acquired for various eye angles. By setting the calculated calibration value as calibration (CAL) data, more accurate gaze input can be performed without requiring the user to perform calibration every time they use gaze input. Note that in this embodiment, the calibration value calculated based on the acquired gaze data is correlated with the orientation of the digital camera 100 and is stored and set as CAL data.

[0073] In this embodiment, a total of five gaze points are displayed in calibration mode. The five gaze points are displayed one by one. When gaze data at the first gaze point is acquired, the first gaze point is hidden, the second gaze point is displayed, and so on. Calibration is complete when all gaze data is acquired and the calibration value is calculated.

[0074] The touch panel 70a and the display unit 28 can be integrally constructed. For example, the touch panel 70a is disposed on top of the display surface of the display unit 28, such that the touch panel 70a has sufficiently high light transmittance so as not to interfere with the display of the display unit 28. The input coordinates on the touch panel 70a and the display coordinates on the display screen of the display unit 28 are correlated with each other. This allows for the provision of an operable display object (graphical user interface) as if the user could directly interact with the display screen shown on the display unit 28. The system control unit 50 can detect the operation or status on the touch panel 70a as described below.

[0075] • A finger or pen that has not previously touched touch panel 70a makes a new touch on touch panel 70a. This is referred to as the start of a touch (hereinafter referred to as touch-down).

[0076] • The touch panel 70a is being touched by a finger or pen (hereinafter referred to as "touch-on").

[0077] • While keeping a finger or pen touching the touch panel 70a, the finger or pen moves on the touch panel 70a (hereinafter referred to as "touch-move").

[0078] • Release your finger or pen from the touch panel 70a. This is called touch end (hereinafter referred to as touch-up).

[0079] • The state of the touch panel 70a when nothing is touching (hereinafter referred to as "touch-off").

[0080] When a touch landing is detected, a touch in progress is also detected simultaneously. After a touch landing is detected, touch in progress is typically detected continuously unless a touch release is detected. When a touch in progress is detected, touch movement is detected. Even when a touch in progress is detected, touch movement will not be detected if the touch position does not change. No touch is detected after all fingers and pens on the touch panel have been touched and released.

[0081] These operations / states, along with the position coordinates of the finger or stylus touching the touch panel 70a, are communicated to the system control unit 50 via an internal bus. The system control unit 50 determines the type of operation (touch operation) to be performed on the touch panel 70a based on the communicated information. For touch movement, the direction of movement of the finger or stylus on the touch panel 70a can also be determined based on changes in position coordinates, targeting the vertical and horizontal movement components on the touch panel 70a. If a touch movement longer than or equal to a predetermined distance is detected, a swipe operation is determined to have occurred. A flick is an operation where a finger touches the touch panel, moves quickly a certain distance while maintaining contact with the touch panel, and then releases. In other words, a flick is a rapid finger flick operation on the touch panel 70a. When a touch movement longer than or equal to a predetermined distance and at a speed greater than or equal to a predetermined value is detected, and then a touch release is detected, a flick is determined to have occurred (determined to have occurred after a swipe operation). A pinch-in is an operation in which multiple points (e.g., two points) are touched simultaneously, and their touch positions are moved closer together. A pinch-out is an operation in which their touch positions are moved apart. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch). The touch panel 70a can be any of various types of touch panels, such as resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, and optical sensor types. Depending on the type, a touch is detected by contact with the touch panel or by bringing a finger or stylus close to the touch panel, and either method can be used.

[0082] The digital camera 100 may include an audio input unit (not shown) configured to detect an audio input signal via a built-in microphone or an audio input device connected to the audio input unit via an audio input terminal, and send the audio input to the system control unit 50. In this case, the system control unit 50 selects the input audio signal as needed, performs analog-to-digital conversion, level optimization processing, specific frequency component reduction processing, etc., on the input audio signal to generate an audio signal.

[0083] Figure 6A and Figure 6B Each example shows a screen displaying settings menu related to the field of view. When the user... Figure 6A When "Detailed View Function Settings" is selected in settings item 601, the screen changes to... Figure 6B The image shown (which is in comparison) Figure 6A The settings menu shown is at a lower level. Figure 6BAn example of a menu screen showing detailed settings for the gaze function is shown. The menu screen for detailed settings for the gaze function includes setting items 602 to 606 as selection options (menu items). By toggling between "Enable" and "Disable" for the gaze input function in setting item 602, one can toggle whether the user's gaze is detected, that is, whether the gaze detection block 160 is enabled. In other words, by toggling between "Enable" and "Disable" for the gaze input function in setting item 602, one sets whether to accept gaze input operations using the user's gaze.

[0084] Setting item 603 is used to configure whether to display a gaze pointer at the gaze position. If set to "Enabled," the user can visually identify the gaze position detected by the gaze detection block 160 based on the direction the user's eyes are looking through the displayed gaze pointer. In this embodiment, the gaze pointer is displayed as such as... Figure 8B The indicator shown is pointer 805. It displays a small circle centered on the line-of-sight position, surrounded by larger circles. Figure 8B In the image, the smaller circle of pointer 805 is filled, and the frame of the larger circle is also filled. Each filled area is semi-transparent, ensuring that the subject is not completely hidden even if the filled area overlaps with the AF frame of the subject or LV image. The area between the boundaries of the small and large circles is not filled, ensuring the visibility of the LV image is unaffected. When the gaze input function is disabled in setting 602, pointer 805 is not displayed regardless of whether the gaze pointer display is set to enabled or disabled.

[0085] Setting item 604 is used to configure whether to determine the line-of-sight position (i.e., perform a line-of-sight determination operation) when the first shutter switch (SW1) 62 is turned on, i.e., when the shutter button 61 is half-pressed. When set to "Enabled", the line-of-sight position is determined to be the position detected by the line-of-sight detection block 160 when the shutter button 61 is half-pressed by the user. When set to "Disabled", the line-of-sight position is not determined when the shutter button 61 is half-pressed.

[0086] When setting 604 is set to "Enabled," the line-of-sight position is not determined when the line-of-sight determination button 82 is pressed, but rather when the first shutter switch 62 is activated by pressing the shutter button 61. For example, consider the case where the user half-presses the shutter button 61 after determining the line-of-sight position by pressing the function button; the line-of-sight position at the half-press of the shutter button 61 differs from the line-of-sight position determined by the function button. In this situation, if setting 604 is set to "Enabled," it is possible to determine the line-of-sight position at the moment the shutter button 61 is half-pressed. That is, it is possible to perform AF operation on a subject located at a different line-of-sight position than the position specified before the shutter button 61 is half-pressed, resulting in poor operability. This inconvenience can be avoided if setting 604 is set to "Disabled."

[0087] When the user selects setting 605, the operation mode switches to Calibration (CAL) mode, and calibration begins to acquire the user's line-of-sight data. See below for reference. Figure 4 The CAL mode processing is described. Calibration is performed on the calibration registration number specified in setting item 606. Upon completion of calibration, CAL data is set and stored in association with the registration number. The registration number specified in setting item 606 is a number used to classify and register data among multiple CAL data entries. Even when a single user uses the gaze input function of the digital camera 100, higher accuracy of gaze input can be achieved by acquiring and setting CAL data separately for each case: whether the user uses the camera with the naked eye or the user wears glasses (contact lenses). When the digital camera 100 is used by multiple users rather than a specific user, higher accuracy of gaze position can be achieved by acquiring and setting CAL data individually for each user. Therefore, in this embodiment, five CAL data entries with registration numbers 1 to 5 can be registered in the digital camera 100. If the user never performs calibration, CAL data is not set and stored in any registration number. As described above, CAL data is stored in association with the orientation of the digital camera 100. Note that CAL data associated with both horizontal and vertical orientations can be set and stored, for example, in registration number "1," but different registration numbers are not assigned to the CAL data associated with the horizontal orientation and the CAL data associated with the vertical orientation. Figure 6B In this setup, the gaze input function is enabled, the gaze pointer display is enabled, and the gaze determination function of SW1 is disabled. That is, the user's gaze position is detected, and the pointer is displayed on the LV image on the EVF 29. In response to the gaze determination operation, the AF box moves to the gaze position and performs AF. However, the AF box does not move to the gaze position when SW1 is pressed.

[0088] For each registration number, the settings menu screen displays whether it has been registered and sets the CAL data. Figure 6B In item 606b, it indicates that no CAL data has been set for registration number "1" indicated by item 606a. If CAL data has been set, a circle is displayed in item 606b, and a crosshair is displayed when no CAL data has been set. However, this is merely an example. As described above, two data points can be set for each registration number, one associated with the vertical orientation of the digital camera 100 and the other associated with the horizontal orientation. The crosshair is displayed only when no CAL data has been set for either the horizontal or vertical orientation of the digital camera 100. Instead of using an icon to display the setting status for the horizontal orientation as in the example above, icons can be used to display the setting status for both the horizontal and vertical orientations separately. The setting status of the CAL data may not be displayed in item 606, but a detailed setting status can be displayed when the display switches from setting item 606 to a sub-setting item.

[0089] Figure 3 A flowchart illustrating the control process that begins when the digital camera 100 is activated (when its power is turned on) is shown. This control process is implemented by the system control unit 50 by loading a program stored in the non-volatile memory 56 into the system memory 52 and executing it.

[0090] In S301, the system control unit 50 determines whether the operating mode is image capture mode. If the current mode is image capture mode, the process proceeds to S302; otherwise, the process proceeds to S303.

[0091] In S302, the system control unit 50 processes the image capture mode. (See later) Figure 5 Describes image capture mode processing.

[0092] In S303, the system control unit 50 determines whether the mode is menu mode. If the mode is menu mode, the process proceeds to S304; otherwise, the process proceeds to S305.

[0093] In S304, the system control unit 50 determines whether the user has already instructed for calibration on the settings menu screen. If a calibration instruction has been issued, the process proceeds to S306; otherwise, the process proceeds to S307. More specifically, for example, when the user selects on the settings menu screen shown in Figure 6... Figure 6B When setting option 605 is selected, it is confirmed that a calibration command has been issued.

[0094] In S305, the system control unit 50 performs other processing. This other processing includes, for example, control processing for playing back captured images, such as processing in playback mode.

[0095] In S306, the system control unit 50 performs calibration (CAL) mode processing. The following will refer to... Figure 4 Describe CAL mode processing.

[0096] In S307, the system control unit 50 performs menu mode processing other than CAL mode processing. Menu mode processing other than CAL mode processing refers to: when selecting a setting item (excluding setting item 605)... Figure 6A or Figure 6B When selecting settings other than those shown in the settings menu on the screen, or when... Figure 6A or Figure 6B The processing is performed when there are settings other than those shown.

[0097] In S308, the system control unit 50 determines whether the processing has been completed. If the processing has been completed, the process ends. Figure 3 The control flow diagram shown is as follows; otherwise, the process returns to S301. The end of the process, for example, means turning off the power to the digital camera 100.

[0098] Figure 4 Is Figure 3 The control flow diagram for CAL mode processing is described in S306. When in Figure 3 When S304 is set to "Yes", that is, when it is determined that an instruction to perform CAL mode processing has been issued, the processing begins. Figures 7A to 7G The image shows an example of the information displayed on EVF 29 in CAL mode processing.

[0099] In S401, the system control unit 50 displays the calibration start screen. Figure 7A The image shows an example of the calibration start screen displayed on the EVF 29. Figure 7A In the middle, guide 701 is displayed to provide tips or suggestions on how to obtain user line-of-sight data with greater accuracy through calibration. When the M-Fn button (in the selection item 702) is included... Figure 1A and Figure 1B When a button (not shown) is pressed by the user, the screen changes to a screen used to obtain user gaze data (such as...). Figures 7B to 7F (The image shown).

[0100] In S402, the system control unit 50 acquires the orientation of the digital camera 100. More specifically, the posture detection unit 55 acquires the orientation of the digital camera 100 relative to the direction of gravity when the selection option 702 displayed on the EVF 29 is pressed in S401, and stores it in the non-volatile memory 56. The orientation of the digital camera 100 acquired in this step is set (stored) as CAL data, such that it is associated with the user's gaze data described below. In this step, it is determined whether the digital camera 100 is horizontally or vertically oriented relative to the direction of gravity. However, this is merely an example. More detailed orientation of the camera can be acquired and stored based on the orientation of the eyepiece 16 and / or the grip 90 relative to the direction of gravity.

[0101] In S403, the system control unit 50 displays the gaze point and acquires the user's gaze data. In this embodiment, as... Figures 7B to 7F As shown, a total of five gaze points are displayed sequentially at different locations on the EVF 29, and the user's gaze data is acquired. Even if the orientation of the digital camera 100 changes during the processing of sequentially displaying a total of five gaze points on the EVF 29, the gaze data is stored in association with the orientation of the digital camera 100 obtained when acquiring the first gaze data. As a specific example, consider the case where the orientation of the digital camera 100 remains horizontal during the acquisition of gaze data at the first three gaze points, but the user changes the orientation to vertical when acquiring gaze data at the fourth gaze point. In this case, although the orientation of the digital camera 100 is vertical when acquiring gaze data at the fourth and subsequent gaze points, the gaze data is stored in association with the camera orientation acquired in S402.

[0102] In S404, the system control unit 50 determines whether the acquisition of user gaze data for all five gaze points has been completed. If the data acquisition is complete, the process continues to S405. However, if the acquisition of data for all five gaze points is not complete, the process proceeds to S411. Note that the completion of gaze data acquisition is defined as... Figures 7B to 7F The acquisition of gaze data for all five gaze points shown has been completed.

[0103] In S405, the system control unit 50 determines the orientation of the digital camera 100 acquired in S402. If the digital camera 100 is horizontally oriented, processing proceeds to S406. However, if the orientation is vertical, processing proceeds to S407. The gaze data acquired in CAL mode processing can be set to CAL data corresponding to the orientation of the digital camera 100. This depends on the relative positional relationship between the gaze detection block 160 located in the eyepiece 16 and the user's eye 161. When the digital camera 100 changes from a horizontal orientation to a vertical orientation, the relative positional relationship between the gaze detection block 160 and the eye 161 also changes. This causes a change in the acquired data of reflected infrared light from the eye 161 detected by the gaze detection block 160. Therefore, if CAL data is set for the horizontal orientation of the digital camera 100 but not for the vertical orientation, if the user attempts to use the gaze input function in a vertical orientation, there is a possibility that a difference will occur between the position the user is currently viewing and the calculated gaze position. In order to calculate the line-of-sight position more accurately, it is desired to acquire and set CAL data for both horizontal and vertical orientations of the digital camera 100.

[0104] In S406, the system control unit 50 stores the orientation of the digital camera 100 (horizontal in this step) and the acquired CAL data in non-volatile memory 56, thus associating the orientation and CAL data with each other. As a result, for example, in Figure 6B The horizontal CAL data for the registration number specified in display item 606a is set. If no calibration is performed for the same registration number in the vertical orientation, the CAL data for the vertical orientation is not set. Note that when CAL data is set for at least one of the horizontal and vertical orientations of the digital camera 100, in Figure 6B At the registration number shown in display item 606a, CAL is set. That is, when CAL data is set for at least one orientation of the digital camera 100, either horizontal or vertical, a circle (〇) is assigned to display item 606b. Figure 6B In this case, no CAL data is set on either the horizontal or vertical orientation of the digital camera 100, so the crosshair (X) is assigned to display item 606b.

[0105] In S407, the system control unit 50 stores the orientation of the digital camera 100 (vertical in this step) and the acquired CAL data in a non-volatile memory 56 in a manner that associates the orientation and CAL data with each other.

[0106] In S408, the system control unit 50 displays a calibration completion screen. An example of the displayed calibration completion screen is shown below. Figure 7GAs shown. By performing calibration multiple times under various conditions and environments, a more accurate gaze position can be detected based on multiple CAL data points. As described above in S405, CAL data can be more accurate when CAL mode processing is performed in both horizontal and vertical orientations of the digital camera 100. For these reasons, a prompt 705 such as "If calibration is performed under various image shooting conditions (in terms of ambient brightness, camera orientation), higher accuracy in gaze detection can be achieved. Would you like to begin calibration?" can be displayed on EVF 29 to prompt the user to perform CAL mode processing from the beginning under different conditions.

[0107] In S409, the system control unit 50 determines whether a command to end calibration has been issued. If a command to end calibration has been issued, the process proceeds to S410. However, if a command to continue calibration has been issued, the process returns to S401. Figure 7G When option 705a is selected on the end screen shown in S406, a command to end calibration is issued. When option 705b is selected, a command to continue calibration is issued. In this case, control processing in CAL mode is performed from the beginning. In this situation, if the orientation of the digital camera 100 changes, gaze data is acquired in the changed orientation, and CAL data is set in association with that orientation. For example, after performing CAL for the horizontal camera orientation of registration number "1", if CAL is then performed for the vertical camera orientation, two datasets (including one dataset obtained in the horizontal orientation and one dataset obtained in the vertical orientation) are stored in registration number "1".

[0108] In S410, the system control unit 50 determines whether the mode is in progress. Figure 4 The CAL mode shown refers to the previous menu mode. If it is menu mode, processing proceeds to S307. However, if it is not menu mode, i.e., in response to a press as described later... Figure 5 When the shortcut button 804a is displayed in S515 and CAL mode processing is started, processing enters S302.

[0109] If the determination in S404 is "No", then in S411, the system control unit 50 determines whether to issue a command to cancel CAL mode processing. If a cancellation command is issued, the processing proceeds to S412; otherwise, the processing returns to S403. A cancellation command can be issued when performing a mode switching operation such as pressing the playback button 79, pressing the shutter button 61, or pressing the menu button 81.

[0110] In S412, the system control unit 50 discards the acquired gaze data (without storing it as CAL data). In this case, the determination in S404 is "No," meaning that gaze data has not yet been acquired for all five gaze points. In this embodiment, since gaze data has not yet been acquired for all five gaze points, it is determined that the high accuracy of the CAL data cannot be guaranteed, therefore the data is not registered as CAL data. In this step, not only the gaze data is discarded, but also the camera orientation obtained in S402 is discarded. After discarding the gaze data in S412, the CAL mode processing ends (and the processing flow returns to...). Figure 3 (S308 shown).

[0111] Figure 5 Is Figure 3 The control flowchart for image capture mode processing described in S302. When in Figure 3 When S301 is set to "Yes", that is, when it is determined that the current mode is the image capture mode, the process begins.

[0112] In the S500, the system control unit 50 displays the live view image captured by the camera unit 22, settings related to the current image shooting mode and image quality, as well as image shooting parameters such as shutter speed and ISO sensitivity on the EVF 29. Figure 8A and Figure 8D An example of displayed information is shown in the image. Figure 8A In the example shown, the digital camera 100 is horizontally oriented, while... Figure 8D In the example shown, the digital camera 100 is vertically oriented. LV 801, image capture information 802, and AF frame 806 are displayed on the EVF 29.

[0113] In S501, the system control unit 50 resets the switching flag (making the switching flag = 0). The switching flag is related to enabling and disabling in... Figure 6A and Figure 6B The flag associated with the switching between gaze input functions specified in setting item 602 is stored in system memory 52. ​​In this embodiment, when the user switches the gaze input function from "disabled" to "enabled" after the digital camera 100 is powered on, the switching flag is set to 1. If the user switches the gaze input function from "enabled" to "disabled", or if the gaze input function remains "disabled" or "enabled" without switching after the power is turned on, the switching flag is 0.

[0114] In S502, the system control unit 50 refers to the system memory 52 and determines whether the line-of-sight input function is enabled (according to the setting specified in setting item 602). If enabled, the process proceeds to S503; otherwise, the process proceeds to S523.

[0115] In S503, the system control unit 50 determines whether to enable... Figure 6B The gaze pointer display setting is specified in setting item 603. If enabled, the process proceeds to S504; if disabled, the process proceeds to S505.

[0116] In S504, the system control unit 50 displays a gaze pointer at the user's gaze position, which is detected by the gaze detection block 160 and calculated considering CAL data (CAL data is not considered if the user has never performed CAL mode processing). The gaze pointer is displayed in a manner that overlays the LV image displayed on the EVF 29. As described above, the gaze pointer is displayed in a manner that... Figure 8B The pointer 805 is displayed in the same way.

[0117] In S505, the system control unit 50 determines whether CAL mode processing has been performed for the current CAL registration number. If CAL mode processing has been performed, the process proceeds to S506; otherwise, it proceeds to S512. In this step, if CAL mode processing has been performed at least once for the current registration number regardless of the orientation of the digital camera 100, it is determined that CAL mode processing has been performed. Figure 6B As shown in display item 606a, the current registration number is displayed on the settings menu screen or on the EVF 29, or on the EVF 29 or on the external viewfinder display unit 43 when the gaze input function is enabled in image capture mode.

[0118] In S506, the system control unit 50 determines the orientation of the housing of the digital camera 100. If the housing is horizontally oriented, the process proceeds to S507, and if the housing is vertically oriented, the process proceeds to S508. The orientation of the housing of the digital camera 100 can be detected by the posture detection unit 55. When the digital camera 100 is horizontal relative to the direction of gravity, that is, in this embodiment, when the eyepiece 16 is perpendicular to the direction of gravity, the digital camera 100 is determined to be horizontally oriented. In this determination, the orientation of the eyepiece 16 relative to the direction of gravity is not considered. When the digital camera 100 is vertical, that is, in this embodiment, when the eyepiece 16 is along the direction of gravity and the grip 90 is located above or below the direction of gravity, the digital camera 100 is determined to be vertically oriented.

[0119] In S507, the system control unit 50 determines whether CAL data has been set for the camera's horizontal orientation. If CAL data has been set for the camera's horizontal orientation, the process proceeds to S509. However, if CAL data has not been set for the horizontal orientation, the process proceeds to S512. In this example, it is determined in S506 that the current orientation of the digital camera 100 is horizontal. That is, if it is determined to be "yes" in this step, the already set CAL data can be used to accurately detect the gaze. On the other hand, if it is determined to be "no" in this step, there is a possibility that when the user holds the digital camera 100 in a horizontal orientation, the gaze position may be displayed at a position different from the position the user is viewing, unless calibration is performed. That is, there is a possibility that there is a difference between the position seen by the user and the detected gaze position, making it difficult for the user to use the gaze input function.

[0120] In S508, the system control unit 50 determines whether CAL data has been set for the camera's vertical orientation. If CAL data has been set for the camera's vertical orientation, the process proceeds to S509; otherwise, the process proceeds to S512. This determination for vertical orientation can be performed in the same manner as the determination for horizontal orientation in S507.

[0121] In S509, the system control unit 50 refers to the system memory 52 to determine whether the switching flag is 1. If the switching flag is 1, the process proceeds to S511; otherwise (if the switching flag is 0), the process proceeds to S510. As described in S501, the switching flag = 1 indicates that the gaze input function is switched from "disabled" to "enabled" after the power to the digital camera 100 is turned on.

[0122] In S510, the system control unit 50 displays icon 803 on EVF 29. Figure 8G An example of the information displayed in this situation is shown. S509 indicates "No": the gaze input function setting has not changed after the digital camera 100 is powered on; that is, the gaze input function is enabled when the digital camera 100 is powered on. In this embodiment, guidance 814 / 824 is displayed whenever the gaze input function setting switches between "enabled" and "disabled," regardless of camera orientation or CAL data settings, so it can be assumed that the user may know whether the gaze input function is enabled or disabled. Therefore, it is assumed that displaying only icon 803 is sufficient, and thus only icon 803 is displayed for notification.

[0123] In S511, the system control unit 50 displays an icon 803 representing an eye shape on the EVF 29 to indicate that the gaze input function is enabled, and displays the text message "Gaze Input: Enabled" (guide 814). An example of the screen displayed in this situation is as follows: Figure 8B and Figure 8E As shown, in S509, a "Yes" indication is determined: the gaze input function has been switched to "Enabled". Therefore, guide 814 notifies the user in a way that is more easily recognizable to the user that the gaze input function has switched from "Disabled" to "Enabled". In this embodiment, even if the gaze input function switches from disabled to enabled after the digital camera 100 is powered on, the switching flag is set to 1 (switching flag = 1), but this is merely an example and not a limitation. For example, only when at the beginning Figure 5 When the gaze input function is switched from "Disabled" to "Enabled" after the image capture mode processing shown, the switching flag is set to 1. In this case, when the user changes the mode to "Enable gaze input function" → "Screen changes to settings menu screen (menu mode operation)" → "Mode changes to image capture mode", no notification is provided to the user via guide 814 (although icon 803 is displayed). That is, guide 814 is not displayed repeatedly, which makes it less cumbersome for users who know that the gaze input function has been switched to "Enabled".

[0124] After icon 803 or guide 814 is displayed in S510 or S511, the process proceeds to S518.

[0125] Since the determination in S505 is "No", the system control unit 50 illuminates icon 803 in S512, causing icon 803 to flash. This makes it easier for the user to visually identify that CAL data has not been set.

[0126] In S513, similar to S509, the system control unit 50 determines whether the switching flag is 1. If the switching flag is 1, the process proceeds to S514; otherwise (switching flag = 0), the process proceeds to S522.

[0127] In S514, as in S506, the system control unit 50 determines the orientation of the digital camera 100. If it is horizontal, the process proceeds to S515, and if it is vertical, the process proceeds to S517.

[0128] In S515, the system control unit 50 displays Guide 804 overlaid on the LV image displayed on EVF 29. More specifically, it displays text information indicating that the gaze input function is enabled (Guide 804c) and a description of the calibration effect (Guide 804d). In addition, it displays text information prompting for CAL mode processing (Guide 804e) and items allowing easy access to CAL mode processing (Shortcut buttons 804a and 804b). Figure 8C An example of the information displayed in this situation is shown. Details of the displayed information will be described later. The indication that S505 is "No" and S513 is "Yes" indicates that the user has never used the gaze input function. Therefore, the user may not know that it is best to perform CAL and set the CAL data in a specific registration number to use the gaze input function. Therefore, a message is displayed to inform the user that higher accuracy of the gaze input function can be achieved by first performing CAL and then using the gaze input function. When the user attempts to use the gaze input function to capture an image in image capture mode, switching to the settings menu screen and performing CAL mode processing may be inconvenient for the user. To address this situation, a shortcut button (shortcut button 804a) is displayed to allow easy switching to CAL mode processing. By selecting shortcut button 804b, the user can proceed to the next operation without performing CAL again. Since power is consumed when the gaze input function is enabled—that is, when the gaze detection block 160 operates—power will be consumed contrary to the user's intention if the user is unaware that the gaze input function is enabled, resulting in a decrease in battery power. Therefore, in addition to icon 803, guide 804 is displayed overlaid on the LV image, allowing users to more clearly identify that the gaze input function is enabled. Guide 804 includes guides 804c, 804d, and 804e, as well as shortcut buttons 804a and 804b. In this embodiment, guide 804 is displayed when the camera is horizontally oriented. Alternatively, only guides 804c to 804e may be displayed without shortcut buttons 804a and 804b.

[0129] In S516, the system control unit 50 determines whether shortcut button 804a has been selected from shortcut buttons 804a and 804b described in S515, that is, whether an instruction to proceed to CAL mode processing has been given. If an instruction to proceed to CAL mode processing has been given, then processing continues. Figure 3 S306 in the process (where CAL mode processing is required). Without a given instruction, i.e., when shortcut button 804b is selected, the process proceeds to S518.

[0130] In S517, the system control unit 50 displays guide 814 overlaid on the LV image displayed on EVF 29. In this step, unlike S515, only "Eye Input Function: Enabled" (guide 814) is displayed, without guides 804d and 804e and shortcut buttons 804a and 804b. Figure 8F Note that the display in this step may be omitted if the user's line of sight determines the operation or if a predetermined time period has elapsed.

[0131] In S518, the system control unit 50 determines whether a predetermined time has elapsed since the display of boot 804 (displayed in S515) or boot 814 (displayed in S517). If the predetermined time has elapsed, the process proceeds to S519; otherwise, the process proceeds to S521.

[0132] In S519, the system control unit 50 determines whether to perform a line-of-sight determination operation. If a line-of-sight determination operation is performed, the process proceeds to S520. If a line-of-sight determination operation is not performed, the process returns to S518.

[0133] In S520, the system control unit 50 hides the guide 804 displayed in S515 or the guide 814 displayed in S517. Icon 803 is not hidden. This is because icon 803 is displayed in a small area, so displaying icon 803 along with other image capture information is unlikely to disturb the user when an image is captured. By continuing to display icon 803 on the EVF 29, the user can always visually identify whether the gaze input function is enabled or disabled. On the other hand, due to... Figures 8A to 8I As shown, guides 804c to 804e and shortcut buttons 804a and 804b are overlaid on the LV image, so their continuous display will reduce the visibility of the LV image.

[0134] In S521, similar to S520, the system control unit 50 hides the boot 804 displayed in S515 or the boot 814 displayed in S517.

[0135] In S522, the system control unit 50 determines whether the gaze input function has been switched from enabled to disabled. If yes, the process proceeds to S523; otherwise, the process proceeds to S527.

[0136] In S523, the system control unit 50 sets the switching flag to 0 and stores it in the system memory 52.

[0137] In S524, the system control unit 50 hides icon 803 displayed on EVF 29, and displays guide 824 "Look-in Input Function: Disabled" on EVF 29, informing the user that the look-in input function has been disabled. Figure 8I An example of the display on the EVF 29 in this scenario is shown. Figure 8I An example of the display when the digital camera 100 is horizontally oriented is shown, and the display when the digital camera 100 is vertically oriented is similar. As described in S518, when a predetermined time has elapsed after the guide 824 is displayed on the EVF 29, the guide 824 displayed in this step is hidden. If an operation to determine the viewing position is performed before the predetermined time has elapsed, the guide 824 is hidden, and the process proceeds to S527.

[0138] If the result is "No" in S502, the system control unit 50 determines in S525 whether the gaze input function has been switched from disabled to enabled. If it has been switched, the process proceeds to S526; otherwise, the process proceeds to S527.

[0139] In S526, the system control unit 50 sets the switching flag to 1 and stores it in the system memory 52. ​​Then, the process returns to S503.

[0140] In S527, similar to S519, the system control unit 50 determines whether to perform the operation of determining the line of sight position. If the line of sight determination operation is performed, the process proceeds to S528; otherwise, the process proceeds to S532.

[0141] In S528, similar to S502, the system control unit 50 determines whether the gaze input function is enabled. If enabled, the process proceeds to S529; otherwise, the process proceeds to S532.

[0142] In S529, the system control unit 50 determines whether a line of sight has been detected. If the line of sight detection block 160 detects the user's line of sight, it determines that a line of sight has been detected, and the process proceeds to S530. If no line of sight has been detected, the process proceeds to S532.

[0143] In S530, the system control unit 50 determines whether a gaze is detected. If a gaze is detected, the process proceeds to S531; otherwise, the process proceeds to S532.

[0144] In S531, the system control unit 50 moves the AF frame to a position corresponding to the user's line of sight position on the EVF 29 detected / calculated by the line of sight detection block 160 when determining the line of sight position in S527, and the system control unit 50 performs the AF operation. Figure 8H An example of a display in this situation is shown. When the pointer 805, which indicates the user's line of sight, is in... Figure 8GWhen performing a line-of-sight determination operation in the position shown, the AF box 806 is moved to the position of pointer 805 (the user's line-of-sight position), and the AF operation is performed.

[0145] In S532, the system control unit 50 determines whether the first shutter switch 62 is on. If it is on, the process proceeds to S533; otherwise, the process proceeds to S541. The state where the first shutter switch 62 is on refers to the state where the shutter button 61 is half-pressed, as described above. That is, it can be assumed that the user is about to take a picture.

[0146] In S533, the system control unit 50 refers to the non-volatile memory 56 and determines the function of determining the line-of-sight position via SW1 (in Figure 6B Whether setting item 604 in the settings is set to "Enabled". If "Enabled", the process proceeds to S534, and if "Disabled", the process proceeds to S535.

[0147] In S534, the system control unit 50 moves the AF frame to a position corresponding to the user's line of sight position on the EVF 29 detected / calculated by the line of sight detection block 160 when the first shutter switch is turned on in S532, and the system control unit 50 performs the AF operation.

[0148] In S535, the system control unit 50 determines whether the focus mode is set to AF mode. If AF mode is set, the process proceeds to S536; otherwise (if MF mode is set), the process proceeds to S538. Switching between AF and MF modes is done via the settings menu screen or by using a switch located outside the lens unit 150.

[0149] In S536, the system control unit 50 performs AF processing based on the position of the display AF frame.

[0150] In S537, the system control unit 50 operates to prepare for other image capture operations such as AE and AWB.

[0151] In S538, the system control unit 50 determines whether the second shutter switch 64 is on. If the second shutter switch 64 is on, that is, if the shutter button 61 is fully pressed, the process proceeds to S539; otherwise, the process proceeds to S540. In this embodiment, it is assumed that an image capture command is issued when the shutter button 61 is fully pressed, but an image capture command can also be issued, for example, when an icon displayed on the touch panel 70a is touched.

[0152] In S539, the system control unit 50 performs a series of steps until the captured image is stored as an image file on the recording medium 200.

[0153] In S540, the system control unit 50 determines whether the first shutter switch 62 remains in the on state. If so, the process proceeds to S534; otherwise, the process proceeds to S541.

[0154] In S541, the system control unit 50 determines whether the image capturing mode has ended. When the image capturing mode ends, Figure 4 The control flow diagram in A ends, and processing returns to... Figure 3 Step S308. If the image capture mode has not yet ended, the process returns to S502. Ending the image capture mode means, for example, switching to the settings menu screen by turning off the power of the digital camera 100 or pressing the menu button 81.

[0155] When it is determined in S529 that a user's gaze has been detected, the AF box can be moved to the position corresponding to the detected gaze, and the AF operation can be performed without determining the gaze in S530. That is, S530 can be skipped, and processing can proceed from S529 to S531.

[0156] Figures 8A to 8I An example of the display given on the EVF 29 is shown, illustrating the settings in image shooting mode processing based on the orientation and CAL data of the digital camera 100 and its registration number.

[0157] Figure 8A This is an example of the display on the EVF 29 when the digital camera 100 is horizontally oriented and the eye-tracking function is disabled. LV 801 is displayed on the EVF 29, and reference is made to the non-volatile memory 56, with image capture information 802 and the AF frame 806 displayed together with LV.

[0158] Figure 8B The following states are displayed: Digital camera 100 is horizontally oriented, gaze input function is enabled, gaze pointer display is enabled, and CAL for horizontal orientation is registered in the current registration number. When digital camera 100 enters image capture standby mode after the gaze input function is enabled following the power-on of digital camera 100, it displays information such as... Figure 8B The display shown. Besides... Figure 8A In addition to the display shown, icon 803 and guide 814 are displayed to indicate that the gaze input function is enabled.

[0159] Figure 8CAn example of the display is shown when the digital camera 100 is horizontally oriented, the gaze input function is enabled, the gaze pointer display is enabled, and the horizontal orientation CAL is not set in the current registration number. In addition to LV 801, image capture information 802, pointer 805, and AF frame 806 displayed on the EVF 29, icon 803 is displayed to make it flash. Then, "Gaze Input Function: Enabled" (guide 804c) is displayed, along with a description of the calibration effect (guide 804d). Furthermore, text prompting switching to CAL mode processing is displayed (guide 804e), along with shortcut buttons 804a and 804b for easy switching and prompting to switch to CAL mode processing. Figure 8C In the state shown, the user has not yet performed horizontal orientation CAL. Therefore, a more visually identifiable notification is given to prompt the user to perform calibration and register CAL data. Note that guide 804 has a more advanced feature than the reference shown later. Figure 8F The described guide 814 has a larger display area and / or a larger number of statements (a larger number of characters). When the digital camera 100 is horizontally oriented, there is a possibility that the user is trying to use the gaze input function for the first time, or that the user is holding the digital camera 100 and accidentally enables the gaze input function. The user may hold the digital camera 100 without intending to start taking pictures immediately, but the user may make various settings to prepare for taking pictures. In this case, even Figure 8C The large area guide 804 shown is overlaid on the LV image, making it unlikely to annoy the user. Furthermore, the flashing icon 803, displayed alongside guide 804, makes it less likely the user will overlook the fact that CAL has not yet been set, and makes switching to CAL mode easier. Although in Figure 8C The statements that lead to 804c through 804e are each surrounded by a dashed line, which is only for illustrative purposes and the dashed line is not actually displayed with the statement.

[0160] Figure 8D An example of the display on the EVF 29 is shown when the digital camera 100 is vertically oriented and the gaze input function is disabled. LV 801 is displayed on the EVF 29, and reference is made to the non-volatile memory 56, with image capture information 802 and the AF frame 806 displayed along with LV. Although the displayed example is generally similar... Figure 8A The example shown is different in that the digital camera 100 is vertically oriented, so the layout of the display changes according to the display area of ​​the EVF 29.

[0161] Figure 8EThe following states are displayed: Digital camera 100 is vertically oriented, gaze input function is enabled, gaze pointer display is enabled, and the vertical orientation CAL is registered in the current registration number. When digital camera 100 enters image capture standby mode after the gaze input function is enabled following the power-on of digital camera 100, it displays information such as... Figure 8E The display shown. Besides... Figure 8D In addition to the display shown, icon 803 and guide 814 are displayed to indicate that the gaze input function is enabled.

[0162] Figure 8F An example of the display is shown when the digital camera 100 is vertically oriented, the gaze input function is enabled, the gaze pointer display is enabled, and the vertical orientation CAL is not set in the current registration number. In addition to LV801, image capture information 802, pointer 805, and AF frame 806 displayed on the EVF 29, a flashing icon 803 and "Gaze Input Function: Enabled" (guide 814) are also displayed. Figure 8F In the state shown, the user has not yet calibrated the vertical orientation. However, because the digital camera 100 is vertically oriented, the user is more likely to check the composition of the entire image or wait for a shutter opportunity compared to when the digital camera 100 is horizontally oriented. Figure 8F The guide 814 shown has a higher performance than Figure 8C The guide 804 shown has a smaller display area and less text (fewer characters). Although the guide display area is small, the flashing of icon 803 lets the user know that CAL data has not yet been set. This makes it easier for the user to see the live view image and less likely to miss the shutter release opportunity.

[0163] Figure 8G The following states are displayed: Digital camera 100 is horizontally oriented, gaze input function is enabled, gaze pointer display is enabled, and the horizontal orientation CAL is registered in the current registration number. When digital camera 100 enters image capture standby mode after the gaze input function is enabled following the power-on of digital camera 100, it displays information such as... Figure 8G The same display is shown. When the digital camera 100 is in vertical orientation and has already registered a CAL in vertical orientation with the current CAL registration number, the same display is made.

[0164] Figure 8HThis diagram illustrates an example of the display on the EVF 29 when the user performs an operation to determine the position of their gaze, with the digital camera 100 in a horizontal orientation, gaze input function enabled, and gaze pointer display enabled. Instead of pointer 805, an AF frame 806 is displayed based on the position of pointer 805. By temporarily hiding pointer 805 in response to the operation of determining the gaze position, the AF frame and subject in the LV image become easier to see. The temporarily hidden pointer 805 is displayed again after a predetermined time has elapsed or when the second shutter switch 64 is closed.

[0165] Figure 8I The following state is displayed: Digital camera 100 is horizontally oriented, gaze input function is disabled, and vertical orientation CAL is already registered in the current registration number. More specifically, this is the state when the gaze input function is switched from enabled to disabled via user operation. In response to... Figure 8H The status indicated is that gaze input is disabled, icon 803 is hidden, and guide 824 is displayed. The displayed guide allows the user to visually recognize that gaze input has been disabled. For example, because the scheduled time period has passed, icon 803 is not displayed even after guide 824 is hidden, so the user can recognize that gaze input is currently disabled.

[0166] Although icon 803 is lit or flashing to indicate whether calibration has been performed, this is merely an example and not a limitation. Alternatively, depending on whether calibration has been performed, the icon's color may be changed, or the icon may be displayed in a different form. Icon 803 is assumed to be in the shape of an eye, but this is only exemplary and not a limitation.

[0167] As described above, in this embodiment, the display is appropriately changed according to the orientation of the digital camera 100. When the digital camera 100 is horizontally oriented and CAL data is not set, if the gaze input function is enabled, the following display (guide 803) is provided to be overlaid on the live view image: an icon indicating that CAL data has not been set (icon 803), information indicating that the gaze input function is enabled (guide 804c), and text information prompting to switch to CAL mode processing (guide 804d). Furthermore, displays facilitating switching to CAL mode processing and prompting for CAL mode processing are shown (guide 804e), as well as items facilitating switching to CAL mode processing (shortcut buttons 804a and 804b). These displays can prompt the user to calibrate and set CAL data, and switch to CAL mode processing without disturbing the user. When the digital camera 100 is vertically oriented and CAL data is not set, when the gaze input function is enabled, only the icon indicating that CAL data is not set and the indication that the gaze input function is enabled are displayed. When the digital camera 100 is in a horizontal orientation, there is a possibility that the user is processing camera settings for taking an image, and the user may not be holding the digital camera 100 with the intention of immediately taking an image. On the other hand, when the digital camera 100 is in a vertical orientation, the user may be framing an image before taking it, or may be waiting for an opportunity to take an image. In this case, if a display prompting the user to perform calibration is superimposed on the live view image, the user may be distracted by the display's guidance and may even miss the shutter release opportunity. Therefore, when the digital camera 100 is in a vertical orientation, the display should be as small as possible while still being perceptible to the user, and should not interfere with image capture. By controlling it in the above manner, a display can be provided at appropriate timings to prompt the user to set functions related to the control of the electronic device.

[0168] The above description of display control based on whether the calibration of a digital camera with gaze input function is set (depending on the setting state) according to this embodiment is exemplary and not limited to the calibration setting state. For electronic devices designed for use in various orientations, if allowing users to set functions related to the control of the electronic device, this application can be applied to techniques that change the content of the display used to notify the user based on the setting and orientation of the electronic device's functions. That is, when the electronic device is in an orientation where the user should be using it while paying attention to the currently displayed content, the guidance displayed to notify the user is limited to the minimum necessary. When the electronic device is allowed to be used in various ways, a display prompting a change of settings is given when the user is using the electronic device without paying attention to the currently displayed content. By controlling in the above manner, a setting prompt can be given at an appropriate time according to the orientation of the electronic device without disturbing the user. As a variation, the application to a smartphone as a mobile terminal is described below.

[0169] (Variant example) Figures 9A to 9F An example of a display is shown when the invention is applied to a mobile terminal such as a smartphone.

[0170] Figure 9A This is a block diagram illustrating an example of the structure of a smartphone 950. A CPU 941, memory 932, non-volatile memory 956, image processing unit 924, display 900, recording medium I / F 918, gesture detection unit 955, communication I / F 939, system timer 953, operation unit 970, and camera unit 922 are connected to an internal bus. The units connected to the internal bus can exchange data with each other via the internal bus.

[0171] The memory 932 is, for example, RAM (volatile memory using semiconductor elements). The CPU 941 uses the memory 932 as its working memory to control various parts of the smartphone 950 according to programs stored, for example, in non-volatile memory 956. The non-volatile memory 956 stores image data, audio data, various programs for operation by the CPU 941, and other data. For example, the non-volatile memory 956 can be implemented using a hard disk (HDD) or ROM.

[0172] Under the control of CPU 941, image processing unit 924 performs various operations on image data stored in non-volatile memory 956 or recording medium 919, image data acquired via communication I / F 939, captured images, etc. Image processing performed by image processing unit 924 includes A / D conversion processing, D / A conversion processing, image data encoding, compression, decoding, enlargement / reduction (sizing), noise reduction, and color conversion processing. Image processing unit 924 can be implemented using dedicated circuit blocks for specific image processing. Depending on the type of image processing, CPU 941 can perform image processing according to a program without using image processing unit 924.

[0173] Under the control of CPU 941, display 900 displays images, GUI (Graphical User Interface) screens, etc. CPU 941 generates display control signals according to a program and controls various parts of smartphone 950 to generate video signals for display on display 900, and outputs the generated video signals to display 900. Display 900 displays images based on the output video signals.

[0174] The operation unit 970 is an input device for accepting user operations, including character input devices such as a keyboard, pointing devices such as a mouse and touch panel, buttons, dial pads, joysticks, touch sensors, touchpads, etc. The touch panel is an input device configured to cover the display 900 in a flat form so as to output coordinate information based on the touch position.

[0175] Recording medium I / F 918 is configured to allow recording medium 919, such as a memory card, CD, DVD, etc., to be inserted into recording medium I / F 918 for reading data from or writing data to the inserted recording medium 919 under the control of CPU 941. Communication I / F 939 is an interface for communicating with external devices, the Internet 940, etc., to send / receive various data such as files and commands. System timer 953 is a timer for measuring time used for various controls and the time of the built-in clock.

[0176] The imaging unit 922 is a camera unit that includes an imaging device (image sensor) implemented using a CCD device, CMOS device, or the like that converts optical images into electrical signals. The imaging unit 922 includes: a lens group (imaging lens) including a zoom lens and a focusing lens, a shutter with an aperture function, an imaging device, an A / D converter configured to convert analog signals output from the imaging device into digital signals, and a barrier covering the imaging system to prevent contamination and damage. The image processing unit 924 performs predetermined pixel interpolation, size adjustment processing such as reduction, and color conversion processing on the data acquired by the imaging unit 922. The CPU 941 performs exposure control, range control, and AWB (automatic white balance) processing based on the calculation results obtained by the image processing unit 924. Display image data captured by the imaging unit 922 and processed by the image processing unit 924 is displayed on the display 900. The digital signal captured by the camera unit 922, converted once by an A / D converter and stored in the memory 102, is converted into an analog signal by a D / A converter and sequentially sent to the display 900, where it is displayed as a live view image (referred to as live view display). Live view can be displayed when the camera is in standby mode for capturing still images, in standby mode for capturing moving images, or while recording moving images, allowing the captured subject image to be displayed substantially in real time. In response to an image capture preparation command issued based on a user operation on the operation unit 970, the CPU 941 controls the camera unit 922 and the image processing unit 924 to initiate operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB processing, etc. In response to the image capture command, the CPU 941 controls the initiation of a series of image capture processes (actual image capture processes), which include: exposure processing, reading signals from the camera device, processing the captured image by the image processing unit 924 to generate an image file, and recording it on the recording medium 919. Image capture commands can be given by the user through operation of the operation unit 970. The camera unit 922 is capable of capturing still images and moving images.

[0177] The posture detection unit 955 detects the posture (orientation) of the smartphone 950 relative to the direction of gravity. Based on the posture detected by the posture detection unit 955, it can be determined whether the image captured by the camera unit 922 was captured by the smartphone 950 held horizontally or vertically. The CPU 941 can add the orientation information corresponding to the posture detected by the posture detection unit 955 to the image file of the image captured by the camera unit 922, or it can rotate the image and record the resulting rotated image. An accelerometer, gyroscope, or the like can be used as the posture detection unit 955.

[0178] Figures 9B to 9E An example of the display that appears on the display 900 when the smartphone 950 is in a specific orientation and the automatic switching to power-saving mode is turned off is shown.

[0179] Figure 9F A settings menu screen related to power saving mode is shown. This settings menu screen is used to change various settings together to allow for reduced power consumption when the remaining battery power of the smartphone 950 is low. When setting option 911a is selected, the power mode is changed to power saving mode from the time of selection (the settings of the smartphone 950 are changed together to reduce power consumption). When setting option 911b is selected, the current settings are maintained and power saving mode is not entered. Note that when setting option 911a is selected, the mode is changed to power saving mode from the time of selection regardless of the remaining battery power. Setting option 912 sets whether to automatically enter power saving mode when the remaining battery power of the smartphone 950 is equal to or less than a predetermined level (e.g., 20%). When setting option 912a is selected, when the remaining battery power is equal to or less than a certain level, it automatically switches to power saving mode without giving the user any notification such as a guide display. When setting option 912b is selected, when the battery power is equal to or less than a certain level, a notification such as a guide is displayed according to the orientation of the smartphone 950. Figure 9F In the settings, power saving mode is set to off, and automatic power saving mode is also set to off. When power saving mode is on, it will switch to power saving mode regardless of the automatic power saving mode setting (power saving mode and automatic power saving mode are exclusive). When power saving mode is off and automatic power saving mode is on, the mode will automatically switch to power saving mode when the remaining battery power reaches a certain level.

[0180] Figure 9B This example illustrates the display on monitor 900 when smartphone 950 is vertically oriented, power saving mode is off, and the remaining battery level is greater than 20%. Monitor 900 displays: monitor 900, an icon 902 indicating the remaining battery level, and the time. Because power saving mode is off and the remaining battery level is greater than 20% (21% in this example), icon 902 is displayed in green as normal. Image 904 and information such as a title and annotation for image 904 are also displayed on monitor 900.

[0181] Figure 9CAn example of what appears on display 900 when smartphone 950 is vertically oriented, automatic power saving mode is off, and the remaining battery level is equal to or less than 20% is shown. In this example, since the remaining battery level is 20%, icon 902 is displayed in red to indicate a low battery warning. In this embodiment, when the remaining battery level reaches 20%, guide 903 is displayed. Guide 903 includes a display showing the effect of switching to power saving mode when the remaining battery level is low (guide 903c), a prompt to set the power saving mode (guide 903d), and power saving mode setting options 903a and 903b. If option 903a is selected, the mode is switched to power saving mode (power saving mode is enabled). If option 903b is selected, the current settings are maintained and power saving mode is not entered. Icon 902 provides a simple warning indicating that the remaining battery level is equal to or less than a predetermined level, and guide 903 prompts to enter power saving mode.

[0182] like Figure 9B and Figure 9C As shown, in most cases, the smartphone 950 is used vertically, and users may use it without any specific intention. Therefore, it is more user-friendly to notify the user of low battery power by displaying a large amount of information (such as guide 903) in a large display area. When the remaining battery power is equal to or less than 20%, if the user continues to use the smartphone 950 while maintaining the current settings and communication settings, the remaining battery power of the smartphone 950 will reach 0 (depleted) in a short time and the smartphone 950 will become unusable. Since the smartphone can be used for a longer period of time by switching to power saving mode instead of continuing to use it with its current settings, guide 903 and option 903a are displayed to prompt the user to enter power saving mode. By making it possible to change the settings of power saving mode through options 903a and 903b without switching to the settings menu screen, the user can easily change the mode without feeling inconvenienced. When the mode is switched to power saving mode, icon 902 is displayed in yellow to indicate that the mode is power saving mode. This indication is consistent with... Figure 9B Normal icons and Figure 9E The low battery icon in the diagram represents different states (modes). The various statements guiding the 903c and 903d are surrounded by dashed lines, but this is for illustrative purposes and the dashed lines are not actually displayed with the statements.

[0183] Figure 9DThis illustration shows an example of what appears on display 900 when smartphone 950 is horizontally oriented, power saving mode is off, and the remaining battery power is greater than 20%. Image 904, icon 902, and time are displayed on display 900. Because power saving mode is off and the remaining battery power is greater than 20% (85% in this example), icon 902 is displayed in a normal view.

[0184] Figure 9E This illustrates an example of what appears on display 900 when smartphone 950 is horizontally oriented, automatic power saving mode is off, and the remaining battery level is equal to or less than 20%. In this case, icon 902 is displayed in red because the remaining battery level is 19%. When smartphone 950 is used horizontally, even if the remaining battery level is equal to or less than 20%, it will not display anything such as... Figure 9C The guide 903 shows a large area, but icon 902 is displayed in a flashing manner or in a different color. Since the user is holding the smartphone 950 with both hands, the user may be focusing on a video image 904 or a website displayed on the display 900. This is achieved by notifying the user only with icon 902 without displaying other visual aids. Figure 9C The guide 903 shown avoids reducing the visibility of the video image 904, thus preventing user distraction. Furthermore, the low battery level and the video image 904 displayed on the monitor 900 can be viewed.

[0185] The display color of icon 902 changes to red to indicate that the remaining battery power is equal to or less than a certain level (20%), but the notification method is not limited to this. Icon 902 may blink, or a box may be displayed in the outermost part of the display area of ​​display 900 and blink. The threshold for remaining battery power is set to 20%, but the threshold is not limited to this. Users can set the threshold to any value on the settings menu screen.

[0186] The aforementioned technology can be applied to situations other than viewing video image 904. For example, it can be applied to situations where the camera (image capture unit 922) on the smartphone 950 is used to capture still or moving images. More specifically, when a user starts the camera while holding the smartphone 950 in a vertical orientation, they are not always intending to take an image. Instead, the user may change settings related to image capture or select an image capture mode (e.g., panorama mode, portrait mode, time-lapse mode, etc.). In these cases, displaying an indication superimposed on the live view image to prompt switching to low power mode is unlikely to bother the user or cause them to miss the shutter release opportunity. On the other hand, when the camera is started while holding the smartphone 950 in a horizontal orientation, the user may be framing an image or waiting for the opportunity to press the shutter button. In such cases, displaying a prompt to switch to low power mode may annoy the user or cause them to miss the shutter release opportunity. For these reasons, when the smartphone 950 is in a vertical orientation, a prompt for the user to make settings is given, but when the smartphone 950 is in a horizontal orientation, this prompt is not given.

[0187] In other words, when the smartphone is in its primary orientation, no prompts related to setting up smartphone control functions are displayed. However, when the smartphone is in its secondary orientation, prompts for function settings are displayed. This allows for timely prompts for settings based on the electronic device's (smartphone's) orientation, without disturbing the user.

[0188] The above assumptions are that the various controls performed by the system control unit 50 can be performed by a single piece of hardware, or by multiple pieces of hardware (e.g., multiple processors or circuits) through shared processing to control the entire device.

[0189] In the above embodiments, examples of the application of the present invention to digital cameras or smartphones have been described, but these are merely examples and not limitations. For example, the present invention can also be applied to the following situations: If the electronic device allows for the setting of functions related to the control of the electronic device, and the electronic device can be used in various orientations of the electronic device's housing, then the present invention can be applied to that electronic device. That is, the present invention can be applied to tablet PCs, PDAs, portable telephone terminals, portable image viewers, etc. The present invention can also be applied to digital photo frames, music players, game consoles, e-book readers, tablet terminals, smartphones, home appliances with displays, in-vehicle equipment, etc.

[0190] (Other embodiments) The present invention can also be implemented by performing the processes described below. That is, software (programs) for implementing one or more functions of the above embodiments are provided to the system or device via a network or various types of storage media, and the program code is read and executed by the computer (CPU, MPU, etc.) of the system or device. In this case, the program and the storage medium storing the program constitute the present invention.

[0191] This invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, claims are appended to disclose the scope of the invention.

[0192] This application claims priority to Japanese Patent Application No. 2020-209579, filed on December 17, 2020, the entire contents of which are incorporated herein by reference.

Claims

1. A camera device, comprising: A gaze detection unit is configured to detect the user's gaze input; A calibration unit is configured to perform calibration on line-of-sight input and set line-of-sight input calibration data; An orientation detection unit is configured to detect the orientation of the camera device; as well as The control unit is configured to perform control such that, when the orientation detection unit detects that the camera device is in a first orientation and no gaze input calibration data has been set, a first display is shown on the screen to prompt for gaze input calibration. The first display includes display items indicating how to set predetermined functions through user operation. When the orientation detection unit detects that the camera device is in a second orientation different from the first orientation, the first display is not shown. Wherein, the first orientation is the horizontal orientation of the imaging device relative to the direction of gravity. The second orientation is the orientation of the camera device that is perpendicular to the direction of gravity. The control unit performs control such that, in the absence of eye-tracking input calibration data, a second display indicating that eye-tracking input calibration data has not been set is displayed regardless of the orientation of the camera device, and a first display prompting for eye-tracking input calibration is also displayed based on the orientation of the camera device.

2. The camera device according to claim 1, wherein, The first display includes a display showing the effect obtained by performing calibration on the line-of-sight input.

3. The camera device according to claim 1, wherein, When the camera device is in the second orientation, the control unit performs control to display a second display to issue a warning about eye input calibration.

4. The camera device according to claim 3, wherein, The second display is an icon display.

5. The camera device according to claim 3, wherein, The first display has a larger display area than the second display.

6. The camera device according to claim 1, wherein, The control unit performs control to display a live view image captured by the camera unit of the camera device on the display, and displays the first display such that the first display is superimposed on the live view image.

7. The camera device according to claim 1, further comprising: Viewfinder, and Viewfinder display The control unit displays the first display on the viewfinder display.

8. The camera device according to claim 1, wherein, When the gaze input function is set to be unused, the control unit performs control such that the first display is not performed even when the camera device is in the first orientation.

9. The camera device according to claim 1, wherein, The calibration unit can set line-of-sight calibration data for the first and second orientations of the camera device, respectively.

10. The camera device according to claim 1, wherein, Also includes: The setting unit is configured to configure whether to use the gaze input function. The control unit performs control such that, when the gaze input function is used in the setting unit, the first display is displayed according to the orientation of the camera device.

11. The camera device according to claim 9, wherein, The control unit performs control such that, when calibration is performed on one of the first orientation and the second orientation and line-of-sight input calibration data is set, the first display is not displayed regardless of whether the camera device is in the first orientation or the second orientation.

12. The camera device according to claim 1, wherein, The first display includes a shortcut button for switching to the gaze input calibration screen where gaze input calibration is performed.

13. The camera device according to claim 1, wherein, The calibration unit is capable of inputting calibration data for each setting of the line of sight in multiple categories, and The control unit performs control such that, without inputting calibration data for the current setting of the category among the plurality of categories, the first display is displayed according to the orientation of the camera device.

14. The camera device according to claim 9, wherein, The control unit performs control such that when the orientation detection unit detects that the camera device is in a first orientation, even though gaze input calibration data has been set, if the gaze input calibration data does not include gaze input calibration data for the first orientation, the first display is displayed.

15. A control method for a camera device, the camera device supporting gaze input function, the control method comprising: Perform calibration on the gaze input and set the gaze input calibration data; Detect the orientation of the camera device; as well as The control is executed such that when the camera device is detected to be in a first orientation and no eye-tracking input calibration data has been set, a first display is shown on the screen to prompt for eye-tracking input calibration. The first display includes a display item indicating how a predetermined function can be set via user operation. However, if the camera device is detected to be in a second orientation different from the first orientation, the first display is not shown. Wherein, the first orientation is the horizontal orientation of the imaging device relative to the direction of gravity. The second orientation is the orientation of the camera device that is perpendicular to the direction of gravity. The execution control enables a second display indicating that no eye-tracking input calibration data has been set, regardless of the orientation of the camera device, to be displayed when no eye-tracking input calibration data has been set, and also enables a first display prompting for eye-tracking input calibration to be displayed, depending on the orientation of the camera device.

16. A computer-readable storage medium storing a program for causing a computer to function as various units of the camera apparatus according to claim 1.

Images