Electronic apparatus, and method for controlling electronic apparatus

Abstract

To efficiently radiate and photograph a radiation range intended by a user.SOLUTION: An electronic apparatus has: estimation means that estimates a visible range of a user in a picked-up image on the basis of eyeball information that is information on the eye of the user who sees the picked-up image; and control means that controls at least any one of a radiation range and a radiation direction of an illumination device that radiates a subject to be photographed on the basis of the visible range estimated by the estimation means.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an electronic device and a method for controlling an electronic device. [Background technology]

[0002] Recent flash devices set the illumination range based on the lens zoom information of the imaging device, allowing for easy shooting with an appropriate illumination range under basic shooting conditions. Furthermore, flash devices can also control the illumination range based on additional information such as electronic zoom, in addition to lens zoom information. Patent Document 1 discloses that in the optical zoom range, the illumination angle of the flash device is controlled in conjunction with focal length information, and in the electronic zoom range, the illumination angle of the flash device is controlled in conjunction with angle-of-view magnification information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-191595 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the illumination range of the flash device determined based on the information about the optical zoom and the electronic zoom is not necessarily the range intended by the user.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an electronic device that can efficiently illuminate and capture an illumination range that a user intends. [Means for solving the problem]

[0006] The electronic device of the present invention is characterized by having an estimation means for estimating the user's visible range in the captured image based on eyeball information, which is information about the user's eyes viewing the captured image, and a control means for controlling at least one of the illumination range and illumination direction of a lighting device that illuminates a subject to be photographed based on the visible range estimated by the estimation means. [Effects of the Invention]

[0007] According to the present invention, it is possible to efficiently illuminate and photograph an illumination range that the user intends. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is an external view of an imaging device according to a first embodiment. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of an imaging device. [Figure 3A] 4 is a flowchart illustrating an irradiation process according to the first embodiment. [Figure 3B] 4 is a flowchart illustrating an irradiation process according to the first embodiment. [Figure 4] 10A and 10B are diagrams illustrating the relationship between eyeball information and an irradiation range. [Figure 5] 10A and 10B are diagrams illustrating changes in the irradiation range and irradiation direction. [Figure 6] FIG. 10 is an external view of a wearable device according to a second embodiment. [Figure 7] FIG. 1 is a block diagram illustrating an example of the configuration of a wearable device. [Figure 8A] 10 is a flowchart illustrating an irradiation process according to the second embodiment. [Figure 8B] 10 is a flowchart illustrating an irradiation process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] <Embodiment 1> Hereinafter, embodiments of the present invention will be described with reference to the drawings. Figures 1(A) and 1(B) are external views of an image capture device 100 as an example of an electronic device to which the present invention can be applied. Figure 1(A) is a perspective view of the image capture device 100 as seen from the front. Figure 1(B) is a perspective view of the image capture device 100 as seen from the back.

[0010] The imaging device 100 has, on its top surface, a shutter button 61, a power switch 72, a mode selector switch 60, a main electronic dial 71, a sub electronic dial 73, a video button 76, and an extra-finder display 43. The shutter button 61 is an operating member for issuing a shooting preparation instruction or a shooting instruction. The power switch 72 is an operating member for switching the power of the imaging device 100 on and off. The mode selector switch 60 is an operating member for switching between various modes. The main electronic dial 71 is a rotary operating member for changing setting values such as shutter speed and aperture. The sub electronic dial 73 is a rotary operating member for moving the selection frame (cursor), scrolling through images, etc. The video button 76 is an operating member for issuing an instruction to start or stop video shooting (recording). The extra-finder display 43 displays various setting values such as shutter speed and aperture.

[0011] The imaging device 100 has a display unit 28, a touch panel 70a, direction keys 74, a SET button 75, an AE lock button 77, a magnification button 78, a playback button 79, a menu button 80, an eyepiece unit 16, and an eyepiece detection unit 57 on the rear surface.

[0012] The display unit 28 displays images and various information. The touch panel 70a is an operation member that detects touch operations on the display surface (touch operation surface) of the display unit 28. The direction keys 74 are an operation member consisting of keys (four-way keys) that can be pressed up, down, left, and right. Processing can be performed according to the position where the direction keys 74 are pressed. The SET button 75 is an operation member that is pressed mainly to confirm a selection item. The AE lock button 77 is an operation member that is pressed to fix the exposure state in a shooting standby state. The enlargement button 78 is an operation member that switches the enlargement mode on and off in the live view display (LV display) in shooting mode. When the enlargement mode is on, the live view image (LV image) is enlarged or reduced by operating the main electronic dial 71. The enlargement button 78 is also used to enlarge the playback image or increase the magnification ratio in playback mode. The playback button 79 is an operation member that switches between shooting mode and playback mode.

[0013] The menu button 80 is an operating member that is pressed to display a menu screen on the display unit 28 that allows various settings to be made. The user can intuitively make various settings on the menu screen displayed on the display unit 28 using the direction keys 74 and the SET button 75. The eyepiece 16 is the eyepiece of the eyepiece finder (a peer-type finder). The user can view an image displayed on the EVF (Electronic View Finder) 29 inside the imaging device 100 through the eyepiece 16. The eyepiece detection unit 57 is a sensor that detects whether the user has placed their eye on the eyepiece 16 of the eyepiece finder.

[0014] The imaging device 100 also has a grip section 90, a terminal cover 40, a communication terminal 10, and a communication terminal 180. The grip section 90 is a holding section formed in a shape that is easy to hold with the right hand when the user holds the imaging device 100. When the imaging device 100 is held by gripping the grip section 90 with the little finger, ring finger, and middle finger of the right hand, the shutter button 61 and the main electronic dial 71 are arranged in positions that can be operated with the index finger of the right hand. In the same state, the sub electronic dial 73 is arranged in a position that can be operated with the thumb of the right hand. The terminal cover 40 is used to cover a communication terminal 181 ( The communication terminal 10 is a terminal for communicating with a lens unit 150 that is detachable from the image capture device 100. The communication terminal 180 is a terminal for communicating with a flash device 300 that is detachable from the image capture device 100.

[0015] Fig. 2 is a block diagram showing an example of the configuration of an imaging device 100 as an electronic device. The lens unit 150 is a lens unit equipped with an interchangeable photographic lens. The lens 103 is usually composed of multiple lenses, but for simplicity's sake, Fig. 2 shows it as a single lens.

[0016] The communication terminal 6 is a communication terminal through which the lens unit 150 communicates with the imaging device 100, and the communication terminal 10 is a communication terminal through which the imaging device 100 communicates with the lens unit 150. The lens unit 150 communicates with the system control unit 50 via these communication terminals 6 and 10. The lens unit 150 controls the aperture 1 via an aperture drive circuit 2 by an internal lens system control circuit 4. The lens unit 150 also adjusts focus by displacing the lens 103 via an AF drive circuit 3 by the lens system control circuit 4.

[0017] The shutter 101 is, for example, a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

[0018] The outside-finder display unit 43 displays various setting values of the image capturing apparatus 100, such as the shutter speed and aperture, via an outside-finder display unit drive circuit 44.

[0019] The imaging unit 22 is an imaging element (imaging sensor) composed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 22 is provided with an A / D converter (not shown), which is used to convert an analog signal output from the imaging unit 22 into a digital signal. The imaging unit 22 captures an image in synchronization with a horizontal synchronization signal and a vertical line synchronization signal output from a timing generator (not shown). The imaging unit 22 outputs one frame of image data as frame data at the cycle of the vertical line synchronization signal. While the event sensor 163 is an event-based vision sensor (an asynchronous event-based sensor), the imaging unit 22 is a synchronous frame-based sensor.

[0020] The image processing unit 24 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data from the imaging unit 22 (A / D converter) or data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows TTL (through-the-lens) AF (autofocus) processing and AE (autoexposure) processing to be performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL AWB (auto white balance) processing based on the arithmetic results. The image processing unit 24 can also perform picture style processing to convert the captured image (image data) into a color image, a monochrome image, or the like.

[0021] The memory control unit 15 controls data transmission and reception between the imaging unit 22, the image processing unit 24, and the memory 32. Output data from the imaging unit 22 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or via the memory control unit 15 without via the image processing unit 24. The memory 32 stores image data obtained by the imaging unit 22 and image data to be displayed on the display unit 28 and the EVF 29. The memory 32 also serves as a memory for image display (video memory). The display image data written to the memory 32 is displayed on the display unit 28 and the EVF 29 via the memory control unit 15.

[0022] The display unit 28 and the EVF 29 perform display on a display device such as an LCD or an organic EL display in response to a signal from the memory control unit 15. By sequentially transferring and displaying image data stored in the memory 32 to the display unit 28 or the EVF 29, it is possible to display a captured image in a through-view format. The through-view format is the same as what is known as a live view display on a typical digital camera, and in the through-view format, the captured image is displayed with almost no delay. By viewing the through-view image, the user can indirectly view the real space.

[0023] The nonvolatile memory 56 is an electrically erasable and recordable memory, such as a Flash-ROM. The nonvolatile memory 56 stores constants and programs for the operation of the system control unit 50. The programs referred to here are programs for executing the processes of various flowcharts, which will be described later.

[0024] The system control unit 50 is a control unit made up of at least one processor or circuit, and controls the entire imaging device 100. The system control unit 50 performs various processes by executing programs recorded in nonvolatile memory 56. The system memory 52 is, for example, a RAM, and the system control unit 50 loads constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like into the system memory 52. The system control unit 50 also performs display control by controlling the memory 32, the display unit 28, the EVF 29, and the like.

[0025] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0026] The operation unit 70 includes various operation members as an input unit that accepts operations from the user (user operations) to instruct various operations to the system control unit 50. The operation unit 70 includes, for example, the shutter button 61 and a touch panel 70a. The operation unit 70 also includes a main electronic dial 71, a sub electronic dial 73, direction keys 74, a SET button 75, an AE lock button 77, a magnification button 78, a playback button 79, and a menu button 80.

[0027] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on when the shutter button 61 is pressed halfway (a shooting preparation command) and generates a first shutter switch signal SW1. When the first shutter switch signal SW1 is generated, the system control unit 50 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0028] In AE processing, the system control unit 50 calculates and sets appropriate aperture value, shutter speed, and ISO sensitivity for shooting based on the difference between the exposure amount calculated based on the currently set aperture value, shutter speed, and ISO sensitivity and a predetermined appropriate exposure amount. The second shutter switch 63 is turned on when the shutter button 61 is fully pressed (shooting instruction) and generates a second shutter switch signal SW2. When the second shutter switch signal SW2 is generated, the system control unit 50 starts a series of shooting processing operations, from reading out a signal from the imaging unit 22 to writing the captured image.

[0029] The touch panel 70a and the display unit 28 can be configured as an integrated unit. For example, the touch panel 70a is configured so that the light transmittance does not interfere with the display of the display unit 28, and is attached to the upper layer of the display surface of the display unit 28. Then, input coordinates on the touch panel 70a are associated with display coordinates on the display screen of the display unit 28. This allows a graphical user interface to be created, as if the user were directly operating the screen displayed on the display unit 28. It is possible to provide a GUI interface.

[0030] The mode selector switch 60 switches the operating mode between a still image capture mode, a video capture mode, etc. The still image capture modes include, for example, an auto capture mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). The still image capture modes also include various scene modes and custom modes that provide capture settings for different capture scenes. The user can directly switch the operating mode to one of these modes using the mode selector switch 60. Alternatively, the user may switch to a list screen of capture modes using the mode selector switch 60, and then use another operating member to select one of the displayed modes to switch the operating mode. Similarly, the video capture mode may include multiple modes.

[0031] The power supply control unit 31 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, and other components, and detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 31 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage to each component for a predetermined period of time. The power supply unit 30 may be a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a lithium-ion battery, or an AC adapter.

[0032] The communication unit 54 transmits and receives various data such as image signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 54 can also connect to a wireless LAN (Local Area Network) and the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) and Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and can receive image data and various other information from external devices.

[0033] The attitude detection unit 55 detects the attitude of the image capture device 100 with respect to the direction of gravity. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. The attitude detection unit 55 can detect the movement of the image capture device 100 (pan, tilt, roll, whether it is stationary, etc.).

[0034] The eyepiece detection unit 57 is an eyepiece detection sensor that detects the approach (eye) of an object (eye) 160 to and the departure (eye) of the object (eye) 160 from the eyepiece unit 16 of the eyepiece finder. The system control unit 50 switches the display unit 28 and the EVF 29 on and off depending on the state detected by the eyepiece detection unit 57. Change.

[0035] The subject identification unit 59 analyzes the image data obtained by the imaging unit 22 and identifies the subject to be photographed. Specifically, the subject identification unit 59 identifies the type of subject. The subject identification unit 59 can identify the size and position of the subject in the image. The subject identification unit 59 can identify the subject using, for example, a convolutional neural network that is commonly used for image recognition.

[0036] Eyeball detection unit 161 includes eyeball detection lens 162, event sensor 163, and event data calculation unit 164. Eyeball detection unit 161 is capable of detecting information about eye 160 of a user looking through the viewfinder (hereinafter referred to as eyeball information). Infrared light emitted from infrared light-emitting diode 58 is reflected by user's eye 160, and the reflected infrared light passes through eyeball detection lens 162 and forms an image on the imaging surface of event sensor 163.

[0037] The event sensor 163 is an event-based vision sensor that detects changes in the brightness of light incident on each pixel and outputs information about pixels where a brightness change has occurred asynchronously with other pixels. The data output from the event sensor 163 includes, for example, the position coordinates of the pixel where the brightness change (event) occurred, the polarity (positive or negative) of the brightness change, and timing information corresponding to the time the event occurred. This data will hereinafter be referred to as event data.

[0038] Compared to a synchronous frame-based sensor such as the imaging unit 22, the event sensor 163 has reduced redundancy in the information it outputs and is characterized by high-speed operation, a high dynamic range, and low power consumption. On the other hand, since event data (information on pixels where brightness changes have occurred) is output asynchronously with other pixels, processing is performed to determine the relationship between the event data. In order to determine the relationship between the event data, the event data output from the event sensor 163 over a predetermined period of time is accumulated, and various calculation processes are performed on the results.

[0039] The event data calculation unit 164 is a calculation unit for acquiring (detecting) eye information based on event data that is continuously and asynchronously output from the event sensor 163. For example, the event data calculation unit 164 accumulates event data that occurs over a predetermined period of time and processes the data as a set of data to acquire eye information. By changing the accumulation time for accumulating the event data, it is possible to acquire a plurality of pieces of eye information that occur at different speeds.

[0040] The eyeball information includes, for example, gaze position information regarding gaze position (the position where the user is looking), saccade information regarding the direction and speed of saccades, and microsaccade information regarding the frequency and amplitude of microsaccades (amount of change in gaze position). The eyeball information may also include information regarding eye movement other than saccade information and microsaccade information, pupil information regarding pupil size and changes in pupil diameter, and blink information regarding the speed and number of blinks.

[0041] The eyeball information that can be acquired is not limited to the information exemplified above. The event data calculation unit 164 may map the event data for the accumulation time as one frame of image data based on the event occurrence coordinates (the position coordinates of the pixel where the luminance change (event) occurred), and perform image processing. The event data calculation unit 164 can acquire eyeball information by frame-based image processing from one frame of image data obtained by mapping the event data for the accumulation time.

[0042] The user state determination unit 165 is a determination unit that determines the state of the user based on the eyeball information obtained by the event data calculation unit 164. The user state determination unit 165 can acquire the user's visual range from eyeball information such as the frequency and amplitude of microsaccades. The visual range is the range the user is looking at, i.e., the range the user's gaze is directed at, and is synonymous with the attention range or focus range. The user state determination unit 165 can also acquire the user's gaze direction from eyeball information such as gaze position information and saccade direction.

[0043] The user state determination unit 165 can determine the user state as the degree of gaze or the degree of overhead view from, for example, the frequency and amplitude of microsaccades. The degree of gaze is an index that is higher the narrower the visible range and lower the wider the visible range. Contrary to the degree of gaze, the degree of overhead view is an index that is lower the narrower the visible range and higher the wider the visible range.

[0044] Furthermore, the user state determination unit 165 determines the frequency of occurrence of microsaccades, The user's concentration level (degree of concentration) or fatigue level can be determined from the amplitude of the microsaccades, pupil size, change in pupil diameter, blink speed, number of blinks, etc. Fatigue level is an index opposite to concentration level. Furthermore, the user state determination unit 165 can determine preference level from the speed of microsaccades, change in pupil diameter, etc. Preference level is an index that increases when the user is looking at an object that matches the user's preference (such as a favorite face), and decreases when the user is looking at an object that does not match the user's preference, for example.

[0045] Information relating to the user's state, such as the gaze range, gaze intensity (or bird's-eye view), concentration level (or fatigue level), and preference level, will hereinafter be referred to as user state information. The user state determination unit 165 can determine the user state using a machine learning model such as a neural network that receives, as input, parameters relating to eye information and the identification result of the object identification unit 59, and outputs user state information. Note that the user state determination unit 165 is not limited to the above, and can acquire information relating to the user's state by arbitrarily combining information included in the eye information.

[0046] The system control unit 50 can obtain information about the position and size of the area of the subject displayed on the EVF 29 (hereinafter referred to as the subject range). In addition, the eyeball detection unit 161 can obtain information about the range of the EVF 29 in which the user's gaze is directed. Therefore, the system control unit 50 can determine which area of the subject the user is looking at.

[0047] Flash device 300 is an illumination device that illuminates a subject to be photographed, and includes a Xe (xenon) tube light-emitting device, an LED light, etc. Flash device 300 is detachable from imaging device 100. Flash device 300 has a communication terminal 301, a light-emitting unit 303, a system control unit 302, a zoom driving unit 304, and a bounce driving unit 305.

[0048] The communication terminal 301 is a communication terminal through which the flash device 300 communicates with the imaging device 100. The flash device 300 communicates with the system control unit 50 via the communication terminal 301 and the communication terminal 180, and the system control unit 302 controls the light emission of the light emitting unit 303. The system control unit 302 also controls the zoom driving unit 304 to change the irradiation range of the light emitting unit 303, and controls the bounce driving unit 305 to change the irradiation direction of the light emitting unit 303.

[0049] It should be noted that flash device 300 is not limited to a lighting device that is detachable from image capture device 100, but may also be an external lighting device that illuminates a subject to be photographed. The external lighting device is communicably connected to image capture device 100 via wire or wirelessly, and can receive information from image capture device 100 that is used to control the illumination range and illumination direction.

[0050] The external device 400 is an external device that is detachable from the imaging apparatus 100 and may be connected to the imaging apparatus 100 via a connection cable. The communication terminal 401 is a communication terminal through which the external device 400 communicates with the imaging apparatus 100. The external device 400 communicates with the system control unit 50 via the communication terminal 401 and the communication terminal 181. The operation unit 403 of the external device 400 includes various operation members as an input unit that accepts operations from the user. The operation unit 403 includes at least one of a shutter button, a power switch, an electronic dial, a directional key, a video button, a playback button, and a menu read button. The system control unit 402 of the external device 400 communicates with the system control unit 50 and can control the imaging apparatus 100. The system control unit 402 can also control the system control unit 302 of the flash device 300 via the system control unit 50.

[0051] 3A, 3B, 4, and 5, the imaging device 100 according to the first embodiment estimates the user's visible range and controls the illumination range and illumination direction of the flash device 300. I will explain the reasoning.

[0052] 3A and 3B are flowcharts illustrating the illumination process according to embodiment 1. Each process in the flowcharts of FIGS. 3A and 3B is realized by the system control unit 50 of the imaging device 100 loading a program stored in the nonvolatile memory 56 into the system memory 52, executing the program, and controlling each component (block). The process shown in FIGS. 3A and 3B starts when the power switch 72 is turned on and the user's eye 160 approaches the eyepiece 16 of the viewfinder in still image capture mode.

[0053] In step S101, the system control unit 50 acquires a captured image (data of the captured image) captured by the imaging unit 22 and stores it in the video memory area of the memory 32. In step S102, the system control unit 50 reads the captured image from the video memory area of the memory 32 and displays it on the EVF 29.

[0054] In step S103, the system control unit 50 transmits the captured image acquired in step S101 to the subject identification unit 59. The subject identification unit 59 analyzes the received captured image to identify (detect) the subject and acquire information about the subject. The subject information includes the type of subject, the size of the subject in the captured image, the position of the subject in the captured image, etc. The subject identification unit 59 identifies the type of subject and specifies the size and position of the subject in the captured image.

[0055] In step S104, the system control unit 50 determines whether or not the first shutter switch signal SW1 has been detected. The first shutter switch signal SW1 is generated when the user half-presses the shutter button 61 to instruct shooting. If the user operates the shutter button 61 and detects the first shutter switch signal SW1, the system control unit 50 determines that the user intends to shoot. If the system control unit 50 detects the first shutter switch signal SW1, the system control unit 50 proceeds to step S105 and acquires eye information. If the user does not operate the shutter button 61 and does not detect the first shutter switch signal SW1, the system control unit 50 determines that the user does not intend to shoot. If the system control unit 50 does not detect the first shutter switch signal SW1, the system control unit 50 returns to step S101.

[0056] In step S105, the system control unit 50 acquires event data using the event sensor 163. The system control unit 50 transmits the acquired event data to the event data calculation unit 164.

[0057] In step S106, the system control unit 50 analyzes the event data using the event data calculation unit 164, and acquires eyeball information, which is information about the eyes of the user viewing the captured image. The system control unit 50 transmits the acquired eyeball information to the user state determination unit 165 via the memory 32. The eyeball information includes at least one of gaze position information, saccade direction, saccade speed, microsaccade occurrence frequency, microsaccade amplitude, pupil size, change in pupil diameter, blink speed, and blink count.

[0058] In step S107, the system control unit 50 acquires information about the user's visible range and viewing direction using the user state determination unit 165. The system control unit 50 stores the acquired information about the visible range and viewing direction in the memory 32. The information about the line of sight direction may include, for example, the coordinates of the center point of the visible range.

[0059] The relationship between eyeball information and the visible range will be described with reference to Figures 4(A) to 4(D). Figures 4(A) and 4(B) are diagrams showing microsaccade information acquired by the event data calculation unit 164 as eyeball information. The vertical axis represents the position of the pupil center in a coordinate system with the eyeball center as the origin. The horizontal axis is time.

[0060] Figure 4(A) shows a microsaccade waveform when the visual range is wider than the example in Figure 4(B). The graph in the area surrounded by dashed line 411 shows the change in eye position when a microsaccade occurs. The wider the visual range, the larger the amplitude of the eye movement (the amplitude of the eye position) and the higher the oscillatory tendency (the lower the attenuation rate). Furthermore, the frequency of microsaccades during period 412 tends to be higher.

[0061] On the other hand, Figure 4(B) shows a microsaccade waveform when the visual range is narrower than the example in Figure 4(A). The graph in the area surrounded by dashed line 413 shows the change in eye position when a microsaccade occurs. The narrower the visual range, the smaller the amplitude of the eye movement tends to be and the lower the oscillatory nature (the higher the attenuation rate). Furthermore, the frequency of microsaccades during period 414, which has the same length as period 412, tends to be lower.

[0062] The user state determination unit 165 can estimate the user's visible range in the captured image using a machine learning model such as a neural network that uses amplitude and oscillatory characteristics, which are characteristic quantities of eye movement, as input parameters and the visible range as output.

[0063] In step S108, the system control unit 50 determines whether the user's visible range has changed within a predetermined time. If the visible range has not changed within the predetermined time, the process proceeds to step S109. If the visible range has changed within the predetermined time, the process returns to step S105, and the system control unit 50 acquires eyeball information again. Note that even if the visible range has not changed within the predetermined time, the system control unit 50 can determine whether the visible range has changed by continuing to execute the processes of steps S105 to S108.

[0064] In step S109, the system control unit 50 determines whether the visible range acquired in step S107 is wider than a predetermined range. For example, the system control unit 50 determines whether the size of the visible range is larger than a predetermined threshold value TH. The threshold value TH may be a predetermined value or may be determined based on the information about the subject acquired in step S103. For example, the threshold value TH may be the size of the subject range, which is an area including the subject detected from the captured image.

[0065] If the size of the visible range is larger than the threshold TH, the system control unit 50 determines that the user is looking down on the entire angle of view. If the size of the visible range is larger than the threshold TH, the system control unit 50 proceeds to step S110 to acquire the illumination range based on the lens information. If the size of the visible range is smaller than the threshold TH, the system control unit 50 determines that the user is viewing a specific subject. If the size of the visible range is smaller than the threshold TH, the system control unit 50 proceeds to step S111 to acquire the illumination range based on the visible range. If the size of the visible range is equal to the threshold TH, the system control unit 50 may proceed to either step S110 or step S111.

[0066] In step S110, the system controller 50 acquires lens information of the lens unit 150. The lens information includes, for example, zoom information of the lens 103. The system controller 50 acquires the illumination range of the flash device 300 based on the lens information. The illumination direction of the flash device 300 can be set to a preset direction.

[0067] In step S111, the system control unit 50 acquires the illumination range of the flash device 300 based on the visible range acquired in step S107. In step S112, the system control unit 50 acquires the illumination direction of the flash device 300 based on the line of sight direction acquired in step S107.

[0068] An example of the method for determining the illumination range and illumination direction described in steps S109 to S112 will be described with reference to Figures 4(C) and 4(D). Figures 4(C) and 4(D) correspond to the microsaccade waveforms of Figures 4(A) and 4(B), respectively, and are diagrams illustrating the user's visual state on the EVF 29. Subject 415 is the subject to be photographed.

[0069] Subject area 418 is the subject area identified by subject identification section 59 in step S103. In FIGS. 4(C) and 4(D), subject identification section 59 identifies the person's head as subject area 418.

[0070] 4(C) shows an example in which the user's visible range 416 is wider than the subject range 418, and it is determined in step S109 that the size of the visible range 416 is greater than the threshold value TH. FIG. 4(D) shows an example in which the user's visible range 417 is narrower than the subject range 418, and it is determined in step S109 that the size of the visible range 417 is smaller than the threshold value TH. The visible range 416 and the visible range 417 are the visible ranges acquired by the user state determination unit 165 in step S107.

[0071] If the user's viewable range 416 is wider than the subject range 418, as shown in FIG. 4(C), it is estimated that the user is looking down on the subject 415 and the background. In order to capture the image as intended by the user, it is desirable for the flash device 300 to irradiate the entire field of view of the image. In this case, in step S110, the irradiation range of the flash device 300 is set based on the lens information acquired from the lens unit 150. The irradiation direction (angle) of the flash device 300 is a preset direction, and can be, for example, the optical axis direction. Alternatively, the irradiation direction may be the direction of the subject identified by the subject identification unit 59.

[0072] 4(D), when the user's visible range 417 is narrower than the subject range 418, it is estimated that the user is gazing intently at the subject 415. In order to capture the image as intended by the user, it is desirable for the flash device 300 to irradiate the subject range 418. In this case, in steps S111 and S112, the illumination range and illumination direction (angle) of the flash device 300 are set based on the visible range 417.

[0073] In step S113, the system control unit 50 records in the system memory 52 the information on the irradiation range and irradiation direction of the flash device 300 acquired in step S110 or steps S111 and S112.

[0074] 3B, the system control unit 50 stops the process of acquiring eyeball information in steps S105 to S108. Even when the system control unit 50 determines in step S108 that the user's visible range has not changed, the system control unit 50 continues to execute the process of acquiring eyeball information, but stops the process of acquiring eyeball information while information related to the irradiation range and irradiation direction is displayed.

[0075] In step S115, the system control unit 50 superimposes the information about the illumination range and illumination direction of the flash device 300 recorded in step S113 onto the captured image and displays it on at least one of the display unit 28 and the EVF 29. The user can check whether the displayed illumination range and illumination direction of the flash device 300 are the intended illumination range and illumination direction.

[0076] In step S116, the system control unit 50 determines whether the user has changed the illumination range or illumination direction. The system control unit 50 receives a change operation via the operation unit 70 to instruct a change to the illumination range and illumination direction of the flash device 300. The operation may be an operation to instruct a change in at least one of the illumination range and the illumination direction.

[0077] If the user has not changed the illumination range or illumination direction, the system control unit 50 determines that the illumination range and illumination direction displayed in step S115 are the illumination range and illumination direction intended by the user. If the user has not changed the illumination range or illumination direction, the process proceeds to step S119. If the user has changed the illumination range or illumination direction, the system control unit 50 determines that the illumination range and illumination direction displayed in step S115 are different from the illumination range and illumination direction intended by the user. If the user has changed the illumination range or illumination direction, the process proceeds to step S117.

[0078] In step S117, the system control unit 50 determines whether the user has changed and confirmed the irradiation range and irradiation direction. If the changes to the irradiation range and irradiation direction have been confirmed, the process proceeds to step S118. If the changes to the irradiation range and irradiation direction have not been confirmed, the process returns to step S116.

[0079] The change (adjustment) of the illumination range and illumination direction in step S117 will be described with reference to Figures 5(A) and 5(B). Figure 5(A) shows an example of illumination range 501 displayed on the EVF 29 in step S115. Figure 5(B) shows an example of illumination range 504 displayed after the user operates the operation unit 70 to change the illumination range in step S117.

[0080] The illumination range 501 in Fig. 5(A) is the illumination range recorded in the system memory 52 in step S113, and is displayed superimposed on the captured image displayed on the EVF 29. The method of displaying the illumination range is not limited to enclosing it with a line, and may be, for example, a form in which zoom information corresponding to the illumination range is displayed numerically as illumination range information 502. Furthermore, in addition to the illumination range information 502, information on the illumination angle indicating the illumination direction may be displayed numerically or the like. Note that either the illumination range 501 or the illumination range information 502 may be displayed, or both may be displayed as shown in Fig. 5(A).

[0081] The operation member information 503 indicates information about an operation member with which the user can change the illumination range and illumination direction in step S117. In the example of Figures 5(A) and 5(B), the operation member for changing the illumination range is the main electronic dial 71, and the operation member for changing the illumination direction is the direction key 74. Other operation members of the operation unit 70 of the imaging device 100 may be assigned to the operation members for changing the illumination range and illumination direction.

[0082] 5(B) is the illumination range after being changed by the operation member indicated in the operation member information 503. As with the illumination information 502 in FIG. 5(A), the illumination range 504 may be displayed as illumination range information 505, in which zoom information corresponding to the illumination range is displayed numerically.

[0083] The illumination range and illumination direction may be displayed on the display unit 28 instead of on the EVF 29. The user's operation to change the illumination range and illumination direction is not limited to being performed by the operation unit 70, and may be received via the operation unit 403 of an external device 400 connectable to the imaging device 100.

[0084] In step S118, the system control unit 50 records the information on the illumination range and illumination direction of the flash device 300 changed in step S117 in the system memory 52. In step S119, the system control unit 50 instructs the system control unit 302 of the flash device 300 to control the zoom driving unit 304 and bounce driving unit 305 based on the illumination range and illumination direction recorded in the system memory 52. The system control unit 302 In accordance with instructions from the system control unit 50, the zoom driving unit 304 and the bounce driving unit 305 are controlled.

[0085] In step S120, the system control unit 50 determines whether or not the second shutter switch signal SW2 has been detected. The second shutter switch signal SW2 is generated when the user fully presses the shutter button 61 to instruct shooting. If the user operates the shutter button 61 and the system control unit 50 detects the second shutter switch signal SW2, the system control unit 50 proceeds to step S121 to start irradiation. If the user does not operate the shutter button 61 and the system control unit 50 does not detect the second shutter switch signal SW2, the system control unit 50 returns to step S115 to accept another change in the irradiation range.

[0086] In step S121, the system control unit 302 of the flash device 300 causes the light emitting unit 303 to emit light. When the light emitting unit 303 starts emitting light, the imaging device 100 starts reading out a signal from the imaging unit 22.

[0087] According to the above-described embodiment 1, the imaging device 100 controls at least one of the illumination range and illumination direction of the flash device 300 based on the user's visible range, and can capture an image by illuminating the illumination range and illumination direction intended by the user.

[0088] <Embodiment 2> In a second embodiment, a flashlight device (illumination device) of an electronic device according to the present invention has a plurality of light-emitting units each having a different illumination range. The electronic device determines which of the plurality of light-emitting units to control based on the user's visible range and the illumination range of each of the plurality of light-emitting units. The electronic device controls at least one of the illumination range and illumination direction of the flashlight device by controlling the determined light-emitting unit to be controlled.

[0089] Fig. 6 is an external view of a wearable device 1000 as an example of an electronic device to which the present invention can be applied. Fig. 7 is a block diagram showing an example of the configuration of the wearable device 1000. In the wearable device 1000 according to the second embodiment, the same components (including operation members) as those in the imaging device 100 according to the first embodiment are denoted by the same reference numerals, and a description thereof will be omitted. Below, a description will be given of the configuration that differs from the first embodiment.

[0090] 6, the right-eye imaging unit 150R and the left-eye imaging unit 150L each include a lens 103 and an imaging unit 22. The right-eye imaging unit 150R and the left-eye imaging unit 150L each include a zoom mechanism, and the user can change the zoom magnification of the right-eye imaging unit 150R and the left-eye imaging unit 150L using the operation unit 70. The wearable device 110 can perform, as zoom operations, optical zoom, which controls the lens position using the zoom mechanism, electronic zoom, which cuts out and enlarges a portion of a captured image, or zoom processing that combines optical zoom and electronic zoom.

[0091] The right-eye display unit 160R and the left-eye display unit 160L each include an eyepiece 16, an EVF 29, and an eyeball detection unit 161. When the user wears the wearable device 110, the user views with their right eye an image displayed on the right-eye display unit 160R (right-eye image) and with their left eye an image displayed on the left-eye display unit 160L (left-eye image). For example, the right-eye display unit 160R displays an image captured by the right-eye imaging unit 150R, and the left-eye display unit 160L displays an image captured by the left-eye imaging unit 150L.

[0092] In FIG. 7, the operation unit 70 includes various operation members as an input unit that accepts operations from the user (user operations). The operation unit 70 includes, for example, an audio UI and a touchpad. The touchpad is mounted on the side (not shown) of the wearable device 1000. The system control unit 50 can detect operations on the touchpad or the state of the touchpad. The coordinates of the position where a finger touches the touchpad are notified to the system control unit 50 via an internal bus, and the system control unit 50 determines what kind of operation (touch operation) has been performed on the touchpad based on the notified information. The touchpad may use any of various touch panel types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or an optical sensor type.

[0093] The power switch 72 is an operating member that switches the power of the wearable device 1000 on and off. The communication unit 54 transmits and receives video signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 54 can transmit and receive data to and from an external database server 18.

[0094] The orientation detection unit 55 detects the orientation of the wearable device 1000 with respect to the direction of gravity. An acceleration sensor, a gyro sensor, or the like can be used as the orientation detection unit 55. The orientation detection unit 55 can detect the movement of the wearable device 1000 (pan, tilt, roll, whether or not it is stationary, etc.).

[0095] The eye proximity detection unit 57 is a wearing detection sensor that detects whether or not the user is wearing the wearable device 1000. The system control unit 50 can switch between starting (power on) and stopping (power off) the wearable device 1000 depending on the state detected by the eye proximity detection unit 57.

[0096] The flash device of wearable device 1000 includes light emitting unit 303R, light emitting unit 303L, zoom driving unit 304R, zoom driving unit 304L, bounce driving unit 305R, and bounce driving unit 305L. Light emitting unit 303R and light emitting unit 303L are, for example, LED lights.

[0097] The system control unit 50 controls the zoom driving unit 304R and the zoom driving unit 304L to control the irradiation ranges of the light-emitting units 303R and 303L, respectively. Furthermore, the system control unit 50 controls the bounce driving unit 305R and the bounce driving unit 305L to control the irradiation directions of the light-emitting units 303R and 303L, respectively.

[0098] The zoom driver 304R is capable of adjusting the zoom position on the telephoto side, and is capable of controlling light emission with a narrower illumination range and higher emission intensity than the zoom driver 304L. The zoom driver 304L is capable of adjusting the zoom position on the wide side, and is capable of controlling light emission with a wider illumination range and lower emission intensity than the zoom driver 304R.

[0099] 8A and 8B, a process of estimating the user's line of sight range and controlling the illumination range and illumination direction of the flash unit (light emitting unit 303R, light emitting unit 303L) using wearable device 1000 according to the second embodiment will be described.

[0100] 8A and 8B are flowcharts illustrating the irradiation process according to the second embodiment. Each process in the flowcharts of FIGS. 8A and 8B is realized by the system control unit 50 of the wearable device 1000 expanding a program stored in the nonvolatile memory 56 into the system memory 52, executing the program, and controlling each component (block). The process shown in FIGS. 8A and 8B is performed when the user wears the wearable device 1000 and the power switch 72 is turned on. It starts by:

[0101] The processing in steps S201 to S203 and the processing in steps S204 to S207 are the same as the processing in steps S101 to S103 and the processing in steps S105 to S108 in FIG. 3A, respectively, and therefore will not be described again.

[0102] In step S208, the system control unit 50 determines whether the visible range acquired in step S206 is wider than a predetermined range. For example, the system control unit 50 determines whether the size of the visible range is larger than a predetermined threshold value TH.

[0103] For example, the threshold value TH is determined based on the maximum value (max value) of the irradiation range at the zoom position that can be set by the zoom driver 304R, or the minimum value (min value) of the irradiation range at the zoom position that can be set by the zoom driver 304L. For example, if the settable range of the zoom position of the zoom driver 304R is 80 mm to 200 mm and the settable range of the zoom position of the zoom driver 304L is 14 mm to 80 mm, the threshold value TH is set to a value (size of the irradiation range) based on a zoom position of 80 mm.

[0104] If the size of the visible range is greater than threshold value TH, i.e., if the visible range is within the illumination range at the zoom position of zoom driver 304L, the process proceeds to step S209. If the size of the visible range is smaller than threshold value TH, i.e., if the visible range is within the illumination range at the zoom position of zoom driver 304R, the process proceeds to step S211. If the size of the visible range is equal to threshold value TH, the system control unit 50 may proceed to either step S209 or step S211.

[0105] In step S209, the system control unit 50 acquires the illumination range of the light-emitting unit 303L based on the visible range acquired in step S206. In step S210, the system control unit 50 acquires the illumination direction of the light-emitting unit 303L based on the visible range acquired in step S206.

[0106] In step S211, the system control unit 50 acquires the illumination range of the light-emitting unit 303R based on the visible range acquired in step S206. In step S212, the system control unit 50 acquires the illumination direction of the light-emitting unit 303R based on the visible range acquired in step S206.

[0107] In step S213, the system control unit 50 records in the system memory 52 the information on the irradiation range and irradiation direction of the light-emitting unit 303L or the information on the irradiation range and irradiation direction of the light-emitting unit 303R obtained in steps S209 to S212.

[0108] 8B, the system control unit 50 stops the process of acquiring eyeball information from steps S204 to S207. Even if the system control unit 50 determines in step S207 that the user's visible range has not changed, the system control unit 50 continues to execute the process of acquiring eyeball information, but stops the process of acquiring eyeball information while information related to the irradiation range and irradiation direction is displayed.

[0109] In step S215, the system control unit 50 displays on the EVF 29 the information about the illumination range and illumination direction of the light emitting unit 303R or the light emitting unit 303L recorded in step S213, superimposed on the captured image.

[0110] In step S216, the system control unit 50 determines whether the illumination range or illumination direction has been changed by the user. The light source 300 receives a change operation that instructs a change in the illumination range and illumination direction of the light device 300. The change operation may be an operation that instructs a change in at least one of the illumination range and illumination direction.

[0111] If the user has not changed the illumination range or illumination direction, the system control unit 50 determines that the illumination range and illumination direction displayed in step S215 are the illumination range and illumination direction intended by the user. If the user has not changed the illumination range or illumination direction, the process proceeds to step S219. If the user has changed the illumination range or illumination direction, the system control unit 50 determines that the illumination range and illumination direction displayed in step S115 are different from the illumination range and illumination direction intended by the user. If the user has changed the illumination range or illumination direction, the process proceeds to step S217.

[0112] In step S217, the system control unit 50 determines whether the user has changed and confirmed the irradiation range and irradiation direction. If the changes to the irradiation range and irradiation direction have been confirmed, the process proceeds to step S218. If the changes to the irradiation range and irradiation direction have not been confirmed, the process returns to step S216.

[0113] In step S218, the system control unit 50 records in the system memory 52 the irradiation range or irradiation direction of the light-emitting unit 303R or the light-emitting unit 303L changed in step S217.

[0114] In step S219, the system control unit 50 controls the zoom driving unit 304R or 304L and the bounce driving unit 305R or 305L based on the irradiation range and irradiation direction recorded in the system memory 52.

[0115] In step S219, the system control unit 50 controls the zoom driving unit 304 and bounce driving unit 305 of the light-emitting unit 303 that correspond to the illumination range and illumination direction changed by the user. For example, if information about the illumination range and illumination direction corresponding to light-emitting unit 303R is displayed on the EVF 29 and the illumination range and illumination direction are changed to those corresponding to light-emitting unit 303L, the system control unit 50 controls the zoom driving unit 304L and bounce driving unit 305L. Conversely, if the illumination range and illumination direction of light-emitting unit 303L are displayed on the EVF 29 and the illumination range and illumination direction are changed to those corresponding to light-emitting unit 303R, the system control unit 50 controls the zoom driving unit 304R and bounce driving unit 305R.

[0116] In step S220, the system control unit 50 determines whether or not a light emission instruction operation has been detected. The system control unit 50 can determine that a light emission instruction operation has been detected when an operation on an operating member to which the light emission instruction operation has been assigned has been detected. If a light emission instruction operation has been detected, the system control unit 50 proceeds to step S221 to start irradiation. If a light emission instruction operation has not been detected, the system control unit 50 returns to step S215 to again accept a change in the irradiation range.

[0117] In step S221, the system control unit 50 causes the light emitting unit 303R or the light emitting unit 303L to emit light based on the control in step S219.

[0118] Although the light-emitting unit 303R has been described as a light-emitting unit for the telephoto side, and the light-emitting unit 303L as a light-emitting unit for the wide side, the light-emitting unit 303R may be assigned to the wide side, and the light-emitting unit 303L may be assigned to the telephoto side.

[0119] The number of light emitting units 303 in the flash device is not limited to two, and the flash device may be provided with three or more light emitting units 303 each having a different illumination range. The system control unit 50 has a zoom drive unit 304 and a bounce drive unit 305. The system control unit 50 determines the light-emitting unit to be controlled based on the user's visible range and the irradiation range of each light-emitting unit 303. The system control unit 50 controls the zoom drive unit 304 and bounce drive unit 305 corresponding to the determined light-emitting unit 303, thereby controlling the irradiation range and irradiation direction of the flash device.

[0120] According to the above-described second embodiment, wearable device 1000 is equipped with a plurality of light-emitting units each having a different illumination range, thereby enabling a wider illumination range to be set compared to a configuration with a single light-emitting unit. Also, by selecting a light-emitting unit suitable for the user's visible range from the plurality of light-emitting units, it becomes possible to improve responsiveness to changes in the eye position and visible range when controlling zoom drive and bounce drive over a wider range from the wide-angle end to the telephoto end.

[0121] In the first and second embodiments, the electronic devices (imaging device 100 and wearable device 1000) estimate the visible range using event sensor 163. Event-based sensors can acquire data with lower power consumption and lower latency than frame-based sensors. However, the present invention is not limited to a configuration in which the visible range is estimated using an event-based sensor. As long as the power consumption and data transfer volume are within an acceptable range, the electronic devices may use a frame-based sensor with a high frame rate.

[0122] The visible range may also be estimated using a combination of an event-based sensor and a frame-based sensor. For example, the electronic device may estimate the visible range by combining a visible range estimated using an event-based sensor with a visible range estimated using a frame-based sensor, or may adopt either of the visible ranges depending on a predetermined condition. The predetermined condition may be, for example, a condition such as a change in the moving speed or position and orientation of the user wearing wearable device 1000.

[0123] In addition, in the first and second embodiments, the electronic device estimates the visible range by utilizing microsaccadic movement. By utilizing microsaccadic movement, the electronic device can estimate the visible range relatively quickly. Note that the electronic device is not limited to using microsaccadic movement, and may record changes in gaze position as history information and estimate the visible range from the changes in gaze position over a predetermined period of time.

[0124] Although the imaging device 100 and the wearable device 1000 have been described in detail as preferred embodiments of the present invention, the present invention is not limited to these specific embodiments. Configurations obtained by appropriately modifying or changing the configurations of the above embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above embodiments are also included in the present invention.

[0125] The present invention can be applied to various electronic devices as long as they have the configurations described in the respective embodiments. For example, the electronic device according to the present invention may be a digital microscope, a display device such as an image viewer with a camera, a personal computer, a PDA (Personal Digital Assistant), a mobile phone, a game console, etc. The electronic device can illuminate a range intended by a user with a flashlight device (illumination device) based on eye information including the user's gaze position.

[0126] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It may also be realized by a circuit (for example, an ASIC) that performs one or more functions.

[0127] The disclosure of this embodiment includes the following configuration, method, program, and medium. (Configuration 1) an estimation means for estimating a visible range of the user in the captured image based on eyeball information, which is information about the eyes of the user viewing the captured image; a control means for controlling at least one of an illumination range and an illumination direction of a lighting device that illuminates a subject to be photographed, based on the visible range estimated by the estimation means; An electronic device comprising: (Configuration 2) The eyeball information includes at least one of gaze position information, saccade direction, saccade speed, microsaccade occurrence frequency, microsaccade amplitude, pupil size, change in pupil diameter, blink speed, and blink count. 2. The electronic device according to configuration 1. (Configuration 3) The estimation means acquires the eyeball information when the user instructs to take a photograph. 3. The electronic device according to configuration 1 or 2. (Configuration 4) The control means When the visible range is narrower than a subject range that is a region including the subject detected from the captured image, controlling at least one of the illumination range and the illumination direction of the lighting device based on the eyeball information; When the visible range is wider than the subject range, at least one of the illumination range and the illumination direction of the illumination device is controlled based on zoom information of a lens that captures the captured image. 4. The electronic device according to any one of configurations 1 to 3. (Configuration 5) The control means When the visible range is narrower than a predetermined range, at least one of the illumination range and the illumination direction of the illumination device is controlled based on the eyeball information; When the visible range is wider than the predetermined range, at least one of the illumination range and the illumination direction of the illumination device is controlled based on zoom information of a lens that captures the captured image. 4. The electronic device according to any one of configurations 1 to 3. (Configuration 6) The control means displays information about the illumination range and the illumination direction of the illumination device on a display unit by superimposing the information on the captured image. 6. The electronic device according to any one of configurations 1 to 5. (Configuration 7) further having an electronic viewfinder; The control means displays information about the illumination range and the illumination direction of the illumination device on the electronic viewfinder by superimposing the information on the captured image. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 8) The estimation means stops acquiring the eyeball information while displaying the information relating to the irradiation range and the irradiation direction. 8. The electronic device according to configuration 6 or 7. (Configuration 9) The control means receives a change operation for instructing a change of the illumination range and the illumination direction of the illumination device. 9. The electronic device according to any one of configurations 1 to 8. (Configuration 10) The control means accepts the change operation via an operation unit of the external device. 10. The electronic device according to configuration 9. (Configuration 11) The control means controls at least one of the illumination range and the illumination direction of the lighting device based on the change operation. 11. The electronic device according to configuration 9 or 10. (Configuration 12) The lighting device includes: A light-emitting portion; a zoom driving unit that controls the illumination range of the light emitting unit; a bounce driving unit that controls the irradiation direction of the light emitting unit; 12. The electronic device according to any one of configurations 1 to 11, comprising: (Configuration 13) The control means controls the illumination range by controlling the zoom drive unit, and controls the illumination direction by controlling the bounce drive unit. 13. The electronic device according to configuration 12. (Configuration 14) the lighting device has a plurality of light-emitting units each having a different illumination range, The control means controls at least one of the illumination range and the illumination direction of the lighting device by controlling one of the plurality of light-emitting units based on the visible range and the illumination-capable range of each of the plurality of light-emitting units. 12. The electronic device according to any one of configurations 1 to 11. (method) an estimation step of estimating a visible range of the user in the captured image based on eyeball information that is information about the eyes of the user viewing the captured image; and a control step of controlling at least one of an illumination range and an illumination direction of a lighting device that illuminates a subject to be photographed, based on the visible range estimated in the estimation step. (program) A program for causing a computer to function as each means of the electronic device according to any one of configurations 1 to 14. (medium) A computer-readable storage medium storing a program for causing a computer to function as each means of the electronic device according to any one of configurations 1 to 14. [Explanation of symbols]

[0128] 100: Imaging device (electronic device), 50: System control unit, 165: User state determination unit, 300: Flash device

Claims

1. an estimation means for estimating a visible range of the user in the captured image based on eyeball information, which is information about the eyes of the user viewing the captured image; a control means for controlling at least one of an illumination range and an illumination direction of a lighting device that illuminates a subject to be photographed, based on the visible range estimated by the estimation means; An electronic device comprising:

2. The eyeball information includes at least one of gaze position information, saccade direction, saccade speed, microsaccade occurrence frequency, microsaccade amplitude, pupil size, change in pupil diameter, blink speed, and blink count.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. The estimation means acquires the eyeball information when the user instructs to take a photograph.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. The control means When the visible range is narrower than a subject range that is a region including the subject detected from the captured image, controlling at least one of the illumination range and the illumination direction of the lighting device based on the eyeball information; When the visible range is wider than the subject range, at least one of the illumination range and the illumination direction of the illumination device is controlled based on zoom information of a lens that captures the captured image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

5. The control means When the visible range is narrower than a predetermined range, at least one of the illumination range and the illumination direction of the illumination device is controlled based on the eyeball information; When the visible range is wider than the predetermined range, at least one of the illumination range and the illumination direction of the illumination device is controlled based on zoom information of a lens that captures the captured image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

6. The control means displays information about the illumination range and the illumination direction of the illumination device on a display unit by superimposing the information on the captured image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

7. further having an electronic viewfinder; The control means displays information about the illumination range and the illumination direction of the illumination device on the electronic viewfinder by superimposing the information on the captured image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

8. The estimation means stops acquiring the eyeball information while displaying the information relating to the irradiation range and the irradiation direction.

7. The electronic device according to claim 6, wherein the electronic device is a semiconductor device.

9. The control means receives a change operation for instructing a change of the illumination range and the illumination direction of the illumination device.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

10. The control means accepts the change operation via an operation unit of the external device.

10. The electronic device according to claim 9.

11. The control means controls at least one of the illumination range and the illumination direction of the lighting device based on the change operation.

10. The electronic device according to claim 9.

12. The lighting device includes: A light-emitting portion; a zoom driving unit that controls the illumination range of the light emitting unit; a bounce driving unit that controls the irradiation direction of the light emitting unit; 2. The electronic device according to claim 1, further comprising:

13. The control means controls the illumination range by controlling the zoom drive unit, and controls the illumination direction by controlling the bounce drive unit.

13. The electronic device according to claim 12.

14. the lighting device has a plurality of light-emitting units each having a different illumination range, The control means controls at least one of the illumination range and the illumination direction of the lighting device by controlling one of the plurality of light-emitting units based on the visible range and the illumination-capable range of each of the plurality of light-emitting units.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

15. an estimation step of estimating a visible range of the user in the captured image based on eyeball information that is information about the eyes of the user viewing the captured image; and a control step of controlling at least one of an illumination range and an illumination direction of a lighting device that illuminates a subject to be photographed, based on the visible range estimated in the estimation step.

16. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 14.

Images