Electronic device, control method for electronic device, and program

Abstract

To provide an electronic device that allows recording of a relationship between event data and frame data in a mode in which the relationship can be easily grasped.SOLUTION: An electronic device comprises: frame data acquisition means that acquires frame data generated at a predetermined frame period; event data acquisition means that detects a change in luminance of light incident on pixels, and continuously acquires and outputs event data being information on pixels experiencing the change in luminance in non-synchronization with other pixels; event data arithmetic means that synchronizes an accumulation start time to start accumulation of the event data with the period of the frame data, takes the accumulation time from the accumulation start time until an accumulation end time as a time that is a fraction of an integer or an integral multiple of the frame period, and extracts event information from the event data accumulated in the accumulation time; and recording means that records the event information in association with the frame data synchronized with the accumulation start time.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to electronic devices, and more particularly to recording and utilizing user gaze information. [Background technology]

[0002] It is known that experts and novices move their eyes differently during various tasks. Quantitatively recording dynamic eye movements, including eye gaze information, of experts in conjunction with the work process can be expected to help novices improve their proficiency.

[0003] For example, Patent Document 1 discloses a means for estimating a subject's athletic performance from dynamic changes in the eyes. It is known that there is a correlation between the width of a person's gaze range and athletic performance or proficiency. Here, the gaze range is synonymous with the attention range or the range of attention. The gaze range can be estimated from the characteristics of eye movement, such as the frequency and amplitude of microsaccades.

[0004] Furthermore, Patent Document 2 discloses a means for recording captured images corresponding to the worker's field of vision in association with the capture time and the work process. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-30491 [Patent Document 2] Japanese Patent Application Publication No. 2019-79144 Summary of the Invention [Problem to be solved by the invention]

[0006] When eyeball information is acquired using an event sensor, it is difficult to grasp the relationship between the event data and image data (frame data) because the event data is not synchronized with the image data.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an electronic device that makes it possible to record the relationship between event data and frame data in a manner that makes it easy to understand the relationship between the event data and frame data. [Means for solving the problem]

[0008] A first aspect of the present invention is frame data acquisition means for acquiring frame data generated at a predetermined frame period; Detects changes in the brightness of light incident on a pixel and acquires event data, which is information about pixels where brightness changes have occurred, asynchronously and continuously from other pixels. Rui a vent data acquisition means; The accumulation start time at which accumulation of the event data is started is synchronized with the period of the frame data, and the accumulation time from the accumulation start time to the accumulation end time is set to an integer fraction or an integer multiple of the frame period, and the event is extracted from the event data accumulated during the accumulation time. event data calculation means for extracting event information; a recording means for recording the event information in association with the frame data synchronized with the accumulation start time; Equipped with 、 the frame data acquisition means is an imaging means, the event data acquisition means acquires the event data from the eyes of a user looking at a display device on which the frame data is displayed; The event data calculation means calculates eyeball information related to the user's eyes from the event data. Characterized by It is an electronic device. A second aspect of the present invention is frame data acquisition means for acquiring frame data generated at a predetermined frame period; an event data acquisition means for detecting a change in luminance of light incident on a pixel and continuously acquiring event data, which is information about a pixel in which a luminance change has occurred, asynchronously with other pixels; an event data calculation means for synchronizing an accumulation start time for starting accumulation of the event data with a period of the frame data, setting an accumulation time from the accumulation start time to an accumulation end time as a time divided by an integer or an integer multiple of the frame period, and extracting event information from the event data accumulated during the accumulation time; a recording means for recording the event information in association with the frame data synchronized with the accumulation start time; Equipped with the frame data acquisition means is an imaging means for imaging a field of view of a user, the event data acquisition means acquires the event data from the user's eyes; The event data calculation means calculates eyeball information related to the user's eyes from the event data. The electronic device is characterized by:

[0009] The present invention three The aspect is 1. A method for controlling an electronic device, comprising: a frame data acquisition step of acquiring frame data generated at a predetermined frame period; Detects changes in the brightness of light incident on a pixel and acquires event data, which is information about pixels where brightness changes have occurred, asynchronously and continuously from other pixels. Rui a vent data acquisition step; an event data calculation step of synchronizing an accumulation start time at which accumulation of the event data is started with a period of the frame data, setting an accumulation time from the accumulation start time to an accumulation end time as a time that is an integer fraction or an integer multiple of the frame period, and extracting event information from the event data accumulated during the accumulation time; a recording step of recording the event information in association with the frame data synchronized with the accumulation start time; Including fruit, the frame data acquisition step acquires the frame data from an imaging means; In the event data acquisition step, the event data is acquired from the eyes of a user who is viewing a display device on which the frame data is displayed; In the event data calculation step, eyeball information relating to the user's eyes is calculated from the event data. The present invention relates to a method for controlling an electronic device.

[0010] The present invention four The aspect is On the computer, a frame data acquisition step of acquiring frame data generated at a predetermined frame period; Detects changes in the brightness of light incident on a pixel and acquires event data, which is information about pixels where brightness changes have occurred, asynchronously and continuously from other pixels. Rui a vent data acquisition step; an event data calculation step of synchronizing an accumulation start time at which accumulation of the event data is started with a period of the frame data, setting an accumulation time from the accumulation start time to an accumulation end time as a time that is an integer fraction or an integer multiple of the frame period, and extracting event information from the event data accumulated during the accumulation time; a recording step of recording the event information in association with the frame data synchronized with the accumulation start time; Run 、 the frame data acquisition step acquires the frame data from an imaging means; In the event data acquisition step, the event data is acquired from the eyes of a user who is viewing a display device on which the frame data is displayed; In the event data calculation step, eyeball information relating to the user's eyes is calculated from the event data. It is a program. [Effects of the Invention]

[0011] According to the present invention, the relationship between event data and frame data can be recorded in a manner that allows easy understanding. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an external view of a digital camera 100 according to an embodiment of the present invention; [Figure 2] 1 is a schematic block diagram showing a digital camera 100 according to an embodiment of the present invention; [Figure 3A] Flowchart for explaining the first embodiment [Figure 3B] Flowchart for explaining the first embodiment [Figure 3C] Flowchart for explaining the first embodiment [Figure 4] 1 is a time chart showing the concept of a method for linking information in a first embodiment; [Figure 5] FIG. 10 is a diagram showing an example of a file structure of video data in the first embodiment; [Figure 6A] FIG. 10 is an explanatory diagram of information associated with a photographed image in the second embodiment; [Figure 6B] FIG. 10 is an explanatory diagram of information associated with a photographed image in the second embodiment; [Figure 6C] FIG. 10 is an explanatory diagram of information associated with a photographed image in the second embodiment; [Figure 6D] FIG. 10 is an explanatory diagram of information associated with a photographed image in the second embodiment; [Figure 6E] FIG. 10 is an explanatory diagram of information associated with a photographed image in the second embodiment; [Figure 6F] FIG. 10 is an explanatory diagram of information associated with a photographed image in the second embodiment; [Figure 7] Flowchart for explaining the second embodiment [Figure 8] FIG. 10 is an explanatory diagram of information associated with a captured still image in the third embodiment. [Figure 9] Flowchart for explaining the third embodiment DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Here, the embodiment will be described based on an example in which the present invention is applied to a digital camera (photography device). However, the electronic device to which the present invention can be applied is not limited to a digital camera, and may be any device.

[0014] FIG. 1A is a front perspective view of digital camera 100, and FIG. 1B is a rear perspective view of digital camera 100. FIG.

[0015] 1A and 1B, display unit 28 is a display unit provided on the back of the camera that displays images and various information. Touch panel 70a can detect touch operations on the display surface (operation surface) of display unit 28. Outside-finder display unit 43 is a display unit provided on the top surface of the camera, and displays various camera settings such as shutter speed and aperture.

[0016] The shutter button 61 is an operation unit used to issue shooting instructions. The mode switch 60 is an operation unit used to switch between various modes. The terminal cover 40 is a cover that protects a connector (not shown) that connects the digital camera 100 to a connection cable for external devices. The main electronic dial 71 is a rotary operation unit included in the operation unit 70. By turning this main electronic dial 71, settings such as shutter speed and aperture can be changed. The power switch 72 is an operation unit used to turn the power of the digital camera 100 on and off. The sub electronic dial 73 is a rotary operation unit included in the operation unit 70. This allows the selection frame to be moved and images to be forwarded. The cross key 74 is included in the operation unit 70 and is a cross key (four-way key) whose up, down, left, and right sections can be pressed. Operations can be performed according to the section of the cross key 74 that is pressed. The SET button 75 is included in the operation unit 70 and is a push button that is mainly used to confirm selections. The video button 76 is used to start and stop video shooting (recording). The AE lock button 77 is included in the operation unit 70 and can fix the exposure state by pressing it while the camera is ready to shoot. The enlarge button 78 is included in the operation unit 70 and is an operation button for turning the enlargement mode on and off in the live view display in shooting mode. By turning the enlargement mode on and operating the main electronic dial 71, the live view image can be enlarged or reduced. In playback mode, the button functions as an enlargement button for enlarging the playback image and increasing the magnification. The playback button 79 is included in the operation unit 70 and is an operation button for switching between shooting mode and playback mode. Pressing the playback button 79 in shooting mode switches to playback mode, and the most recent image recorded on the recording medium 200 can be displayed on the display unit 28. The menu button 80 is included in the operation unit 70 and, by pressing it, displays a menu screen on the display unit 28 on which various settings can be made. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the cross key 74, and the SET button 75. The communication terminal 10 is a communication terminal for the digital camera 100 to communicate with a lens unit 150 (detachable) described later.

[0017] The eyepiece 16 is the eyepiece of an eyepiece finder (a peer-type finder), and the user can view the image displayed on the internal EVF 29 through the eyepiece 16. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether or not the user has placed their eye on the eyepiece 16. The lid 202 is the lid of a slot that stores the recording medium 200.

[0018] Grip unit 90 is a holding unit shaped to be easily gripped in the user's right hand when holding digital camera 100. When digital camera 100 is held by gripping grip unit 90 with the little finger, ring finger, and middle finger of the right hand, shutter button 61 and main electronic dial 71 are positioned so that they can be operated with the index finger of the right hand. In the same state, sub electronic dial 73 is positioned so that it can be operated with the thumb of the right hand.

[0019] FIG. 2 is a block diagram showing an example of the hardware configuration of the digital camera 100 according to this embodiment.

[0020] In Figure 2, lens unit 150 is a lens unit equipped with an interchangeable photographic lens. Lens 103 is usually composed of multiple lenses, but here it is shown as just one lens for simplicity's sake. Communication terminal 6 is a communication terminal that enables lens unit 150 to communicate with digital camera 100. Lens unit 150 can communicate with system control unit 50 via communication terminal 6 and the aforementioned communication terminal 10. Lens unit 150 controls aperture 1 via aperture drive circuit 2 using an internal lens system control circuit 4, and focuses by displacing lens 103 via AF drive circuit 3.

[0021] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

[0022] The imaging unit 22 is an imaging element formed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The A / D converter 23 is used to convert an analog signal output from the imaging unit 22 into a digital signal. The imaging unit 22 captures an image in synchronization with a horizontal synchronization signal and a vertical line synchronization signal output from a timing generator unit (not shown), and outputs one frame of image data as frame data at the period of the vertical line synchronization signal. In contrast to the event sensor 163, which is an asynchronous event-based sensor described below, the imaging unit 22 is a frame-based synchronous sensor. The imaging unit 22 can be regarded as image acquisition means that acquires image data including the user's field of view, and can also be regarded as frame data acquisition means that acquires frame data generated at a predetermined frame period.

[0023] The image processing unit 24 performs predetermined pixel interpolation, resizing (e.g., reduction), and color conversion on data from the A / D converter 23 or data from the memory control unit 15 (described later). The image processing unit 24 also performs predetermined arithmetic processing using captured image data. The system control unit 50 controls exposure and distance measurement based on the arithmetic results obtained by the image processing unit 24. This allows for TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure), and EF (pre-flash) processing. AF modes include a one-shot AF mode that fixes the focus on a predetermined area and a servo AF mode that maintains focus on a predetermined subject. The user can switch between these modes at will by operating the operation unit 70. Automatic switching between these AF modes is also possible (AI focus AF).

[0024] The image processing unit 24 further performs predetermined calculations using the captured image data, and performs TTL-type AWB (auto white balance) processing based on the calculation results. In the case of still image capture, the image data is encoded in a recording format such as JPEG to generate a still image file. In the case of video capture, video data encoded in H.264 / AVC or similar is multiplexed with audio data input from a microphone (not shown), and the multiplexed data is shaped into a video format set as the recording format to generate a video file.

[0025] The memory control unit 15 controls the transmission and reception of data between the A / D converter 23, the image processing unit 24, and the memory 32. The output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, as well as image data to be displayed on the display unit 28 and the EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio. The memory 32 corresponds to a storage means.

[0026] The memory 32 also serves as a memory (video memory) for image display. The display image data written to the memory 32 is displayed on the display unit 28 and EVF 29 via the memory control unit 15. The display unit 28 and EVF 29 display images on displays such as LCDs and organic EL displays in response to signals from the memory control unit 15. Live view display (LV display) can be achieved by sequentially transferring and displaying data that has been A / D converted by the A / D converter 23 and stored in the memory 32 to the display unit 28 or EVF 29. Hereinafter, images displayed in live view are referred to as live view images (LV images). The display unit 28 and EVF 29 have multiple display modes corresponding to the camera's operating modes. For example, there is a shooting mode for LV display, a playback mode for playing back captured images and videos, and a MENU mode for various settings. The user can freely switch between display modes by operating the operation unit 70.

[0027] Various camera settings such as shutter speed and aperture are displayed on the outside viewfinder display 43 via an outside viewfinder display drive circuit 44 .

[0028] The nonvolatile memory 56 is an electrically erasable and recordable memory, and may be, for example, a Flash-ROM. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0029] The system control unit 50, consisting of at least one processor or circuit, controls the entire digital camera 100. It executes programs stored in the nonvolatile memory 56, as described above, to realize the various processes described in this embodiment. The system memory 52, for example, may be a RAM, and stores constants and variables for the system control unit 50, programs read from the nonvolatile memory 56, and other information. The system control unit 50 also controls the memory 32, the display unit 28, and other components to perform display control. Furthermore, the system control unit 50 can detect the camera's operating mode and camera status as work process information. Work process information is information that identifies the user's work. Examples of camera operating modes include still image capture mode SW1, still image capture mode SW2, video capture mode, live view mode, MENU mode, and sleep mode. Examples of camera statuses include tripod-mounted, handheld, walking, running, and panning. The system control unit 50 can be considered a work process information acquisition unit that acquires work process information.

[0030] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0031] The mode selector switch 60, first shutter switch 62, second shutter switch 63, and operation unit 70 are operation means for inputting various operational instructions to the system control unit 50. The mode selector switch 60 switches the operation mode of the system control unit 50 between still image capture mode, video capture mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for specific shooting scenes. The mode selector switch 60 allows the user to directly switch to one of these modes. Alternatively, the user may first switch to a list screen of shooting modes using the mode selector switch 60, then select one of the displayed modes and switch using other operation members. Similarly, the video capture mode may also include multiple modes. The mode selector switch 60 and the system control unit 50 correspond to an operation mode setting means that can set the operation mode of the digital camera 100.

[0032] The first shutter switch 62 is turned on and generates a first shutter switch signal SW1 when the shutter button 61 provided on the digital camera 100 is pressed halfway (a shooting preparation command) during operation. The first shutter switch signal SW1 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0033] The second shutter switch 63 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file, in response to the second shutter switch signal SW2.

[0034] The operation unit 70 is a variety of operation members that serve as an input unit that accepts operations from the user. The operation unit 70 includes at least the following operation parts: the shutter button 61, the touch panel 70a, the main electronic dial 71, the power switch 72, the sub electronic dial 73, the cross key 74, the SET button 75, the movie button 76, the AE lock button 77, the magnification button 78, the playback button 79, and the menu button 80.

[0035] The touch panel 70a and the display unit 28 can be configured as an integrated unit. For example, the touch panel 70a is configured so that its light transmittance does not interfere with the display of the display unit 28, and is attached to the upper layer of the display surface of the display unit 28. Input coordinates on the touch panel 70a are then associated with display coordinates on the display screen of the display unit 28. This makes it possible to provide a GUI (Graphical User Interface) that allows the user to directly operate the screen displayed on the display unit 28. The system control unit 50 can detect the following operations or states on the touch panel 70a: A finger or pen that has not been touching the touch panel 70a touches the touch panel 70a again, that is, the start of touching (hereinafter referred to as touch-down). The touch panel 70a is in a state where it is touched with a finger or a pen (hereinafter referred to as Touch-On). Touching the touch panel 70a with a finger or a pen and moving it (hereinafter referred to as Touch-Move). The finger or pen that has been touching the touch panel 70a is released, that is, the touch ends (hereinafter referred to as "touch-up"). A state where nothing is touching the touch panel 70a (hereinafter referred to as Touch-Off) (called).

[0036] When touch down is detected, touch on is also detected at the same time. After touch down, touch on will usually continue to be detected unless touch up is detected. Touch move is also detected when touch on is detected. Even if touch on is detected, touch move will not be detected unless the touch position moves. Once it is detected that all fingers or pens that were touching have touched up, touch off occurs.

[0037] These operation states and the position coordinates of the finger or pen touching the touch panel 70a are notified to the system control unit 50 via the internal bus. Based on the notified information, the system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 70a. Regarding touch-move, the movement direction of the finger or pen moving on the touch panel 70a can also be determined for each vertical and horizontal component on the touch panel 70a based on changes in the position coordinates. If a touch-move of more than a predetermined distance is detected, a slide operation is performed. A flick is an operation in which a finger is touched to the touch panel, quickly moved a certain distance, and then released. A flick, in other words, is an operation in which a finger is quickly traced across the touch panel 70a as if flicking. When a touch-move is detected over a certain distance or more at a certain speed or more, followed by a touch-up, a flick is determined to have been performed (a flick can be determined to have followed a slide operation). Furthermore, a touch operation in which multiple points (e.g., two points) are touched simultaneously and the touch positions are brought closer together is called a pinch-in, and a touch operation in which the touch positions are moved farther apart is called a pinch-out. Pinch-out and pinch-in are collectively called a pinch operation (or simply a pinch). The touch panel 70a may be any of a variety of touch panels, including resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, and optical sensor types. Depending on the method, there are methods that detect a touch by contact with the touch panel, and methods that detect a touch by the approach of a finger or pen to the touch panel, but either method is acceptable.

[0038] The power supply control unit 31 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the battery type, and the remaining battery power. The power supply control unit 31 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit, including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or Li batteries, an AC adapter, etc.

[0039] The recording medium I / F 17 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0040] The communication unit 54 is connected wirelessly or via a wired cable, and transmits and receives video signals and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also be connected to an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive images and various other information from external devices.

[0041] The orientation detection unit 55 detects the orientation of the digital camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is possible to determine whether an image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. The orientation detection unit 55 can use an acceleration sensor, a gyro sensor, or the like. The orientation detection unit 55 can also detect the movement of the digital camera 100 (panning, tilting, lifting, whether the digital camera 100 is stationary, etc.) using the acceleration sensor or gyro sensor. The orientation detection unit 55 can be considered as a motion information acquisition unit that acquires motion information of the digital camera 100.

[0042] The eyepiece detection unit 57 is an eyepiece detection sensor for detecting (approach detection) the approach (eyepiece approach) and departure (eye separation) of the eyeball (eye) 160 to the eyepiece unit 16 of the finder. The system control unit 50 switches the display unit 28 and the EVF 29 between display (display state) and non-display (non-display state) depending on the state detected by the eyepiece detection unit 57. More specifically, at least in the shooting standby state and when the display destination switching is automatic switching, when the eye is not in contact with the camera, the display is turned on as the display unit 28 and the EVF 29 is not displayed. Also, when the eye is in contact with the camera, the display is turned on as the display unit with the EVF 29 and the display is not displayed as the display unit. The eyepiece detection unit 57 may, for example, An infrared proximity sensor can be used to detect the approach of an object to the eyepiece 16 of the viewfinder incorporating the EVF 29. When an object approaches, infrared light emitted from the infrared-emitting diode 58 is reflected and received by the light-receiving element (not shown) of the infrared proximity sensor. The amount of received infrared light can also determine the distance the object is approaching the eyepiece 16 (eyepiece distance). In this way, the eyepiece detection element 57 performs eyepiece detection, detecting the proximity of an object to the eyepiece 16. When an object approaching within a predetermined distance from the eyepiece 16 is detected from a non-eyepiece state (non-approach state), it is detected as having been approached. When an object detected as approaching from the eyepiece state (approach state) moves away from the eyepiece by more than a predetermined distance, it is detected as having been moved away. The threshold for detecting eyepiece approach and the threshold for detecting eye separation may be different, for example, by providing hysteresis. Furthermore, after eye separation is detected, the eyepiece remains in the eyepiece state until eye separation is detected. After detecting eye separation, the device remains in the non-eye contact state until eye contact is detected. Note that the infrared proximity sensor is just one example, and other sensors may be used for the eye contact detection unit 57 as long as they can detect the approach of an eye or object that can be considered as eye contact.

[0043] The subject identification unit 70 analyzes the image data obtained by the imaging unit 22, identifies the type of subject, and specifies the size and position within the image data. The subject identification unit 70 can use, for example, a convolutional neural network, which is widely used for image recognition.

[0044] Eyeball detection unit 161 is composed of eyeball detection lens 162, event sensor 163, and event data calculation unit 164, which will be described later, and is capable of detecting information (hereinafter referred to as eyeball information) relating to eyeball (eye) 160 of the user looking through the viewfinder. Eyeball detection unit 161 corresponds to eyeball information acquisition means that acquires the user's eyeball information.

[0045] The event sensor 163 detects luminance changes (events) occurring in the eyeball (eye) 160. Infrared light emitted from the infrared-emitting diode 58 is reflected by the eyeball (eye) 160, and the reflected infrared light passes through the eyeball detection lens 162 and forms an image on the imaging surface of the event sensor 163. The event sensor 163 is an event-based vision sensor that detects luminance changes in light incident on each pixel and outputs information about pixels with luminance changes asynchronously with other pixels. The event sensor 163 corresponds to an event data acquisition unit. The data output from the event sensor 163 (hereinafter referred to as event data) includes, for example, the position coordinates of the pixel where the luminance change (event) occurred, the polarity (positive or negative) of the luminance change, and timing information corresponding to the time the event occurred. Compared to a frame-based synchronous sensor such as the existing imaging unit 22, the event sensor 163 eliminates redundancy in the output information and is characterized by high-speed operation, a wide dynamic range, and low power consumption. On the other hand, the event data is information about each pixel output asynchronously. In order to determine the correlation between event data, it is necessary to accumulate event data that occurs over a predetermined period of time and perform various arithmetic operations on the results.

[0046] The event data calculation unit 164 is a calculation unit for detecting eye information (event information) based on the event data continuously and asynchronously output from the event sensor 163. For example, the event data generated over a predetermined period of time is accumulated and processed as a set of data to determine whether or not eye information is present. By changing the accumulation time for accumulating the event data, it is possible to detect multiple pieces of eye information that occur at different speeds.

[0047] The eyeball information includes, for example, eye movement information including at least a part of gaze position information regarding the gaze position of the user, saccade information including the direction and speed of saccades, and microsaccade information including the frequency and amplitude of microsaccades. The eyeball information also includes pupil information including the size of the pupil or the amount of change thereof, and blink information regarding the speed or number of blinks. The above are merely examples of detectable eyeball information, and eyeball information is not limited to these.

[0048] The event data calculation unit 164 may be configured to map the event data for the accumulation time as one frame of image data based on the event occurrence coordinates and perform image processing on the result. This makes it possible to obtain multiple pieces of eye information from the event data using a method that has been used in conventional frame-based image data processing.

[0049] The user state determination unit 165 is a determination unit that determines the state of the user based on the eyeball information detected by the event data calculation unit 164. Examples of the user state include gaze range, gaze degree (overhead degree), concentration level, fatigue level, and preference level. The user state determination unit 165 corresponds to an estimation means that estimates the user state.

[0050] The gaze range or gaze degree (overhead degree) can be determined, for example, from the frequency and amplitude of microsaccades. Here, the gaze range is synonymous with the attention range or focus range, and indicates the range to which attention is being paid. The gaze degree is an index that indicates a higher value as the gaze range is narrower and a lower value as the gaze range is wider. The bird's-eye degree is defined as the antonym of the gaze degree.

[0051] Concentration level can be determined from the frequency and amplitude of microsaccades, pupil size and change, and blink speed and frequency. Concentration level indicates the degree to which the user is concentrating. Fatigue level is defined as the antonym of concentration level.

[0052] Preference is related to the speed of microsaccades and pupil diameter, and can be determined from both parameters. Preference indicates how much a user likes a particular object of gaze (such as a human face).

[0053] The user state determination unit 165 receives input of, for example, parameters related to eyeball information (microsaccades, blinks, pupils) and the identification result of the object identification unit 70, and determines the user state. For this determination, a machine learning-based classifier (learning model) such as a neural network can be used. The classifier can be constructed by machine learning using, as learning data, eyeball information, the object identification result, and indicators representing the user state (hereinafter, referred to as user state information), such as gaze range, gaze level (antonym: bird's-eye view level), concentration level (antonym: fatigue level), and preference level. However, the configuration of the user state determination unit 165 is not limited to the above, and other configurations, such as rule-based determination means, may also be used. The eyeball information and determination results used by the user state determination unit 165 are not limited to the above, and other configurations may also be used.

[0054] The gaze input setting unit 166 enables or disables gaze detection by the eyeball detection unit 161 via the system control unit 50. The gaze input setting unit 166 can also set parameters and detection conditions related to the event data calculation unit 164 and the user state determination unit 165. These can be set arbitrarily by the user using menu settings.

[0055] The system control unit 50 can also obtain information about the area of ​​the EVF 29 in which the subject (object) being photographed is displayed and at what size. Furthermore, the eyeball detection unit 161 can also obtain information about the area of ​​the EVF 29 at which the user is directing their gaze. This allows the system control unit 50 to determine which area of ​​the subject the user is looking at.

[0056] Example 1 Hereinafter, with reference to FIGS. 3A to 3C, 4 and 5, the digital camera according to the first embodiment of the present invention will be described. In this embodiment, the digital camera 100 links and records subject information, eye information of the photographer, and status information of the photographer (user status information) estimated from the correlation between the subject information and the eye information. As described above, the eye information is calculated based on event data accumulated over a predetermined period. In this embodiment, the accumulation start time for starting accumulation of event data is synchronized with the cycle of frame data (image data). Furthermore, the accumulation time from the accumulation start time of the event data to the accumulation end time is set to an integer fraction or an integer multiple of the frame cycle of the frame data.

[0057] Fig. 3 is a flowchart illustrating this embodiment. Each process in the flowchart in Fig. 3 is realized by the system control unit 50 of the digital camera 100 loading a program stored in the nonvolatile memory 56 into the system memory 52, executing it, and controlling each functional block. First, the overall flow will be described with reference to Fig. 3A.

[0058] In step S301, the system control unit 50 detects a user operation via the operation unit 70 and starts video shooting.

[0059] In step S302, the system control unit 50 causes the imaging unit 22 to obtain one frame of image data.

[0060] In step S303, the system control unit 50 increments the frame number of the image data.

[0061] In step S304, the system control unit 50 transmits a display synchronization data signal indicating the image data display timing to the EVF 29 and the event data calculation unit 164. The display synchronization data signal is a data signal that synchronizes timing with a vertical line synchronization signal transmitted to the EVF 29, for example, and includes a frame number.

[0062] In step S305, the system control unit 50 displays on the EVF 29 the image data acquired in step S302.

[0063] In step S306, the system control unit 50 analyzes the frame data (image data) in the subject identification unit 70 and extracts subject information. The subject information includes, for example, any of the type, size information within the frame data, and position information for the subject included in the frame data. The system control unit 50 associates the subject information with the frame number and stores it in the memory 32.

[0064] In step S307, the system control unit 50 calculates the event data obtained from the event sensor 163 using the event data calculation unit 164, and acquires eye information such as gaze position information, saccade information, microsaccade information, pupil information, blink information, etc. The system control unit 50 stores the acquired eye information in the memory 32 in association with the frame number at the time of acquisition of each piece of information.

[0065] In step S308, the system control unit 50 inputs the subject information and eye information to the user state determination unit 165 and obtains various pieces of user state information as output results. At this time, the information input to the user state determination unit 165 is subject information and eye information linked to the same frame number. The system control unit 50 stores the obtained user state information in the memory 32, linking it to the frame number at the time of information acquisition.

[0066] In step S309, the system control unit 50 detects a user operation via the operation unit 70 and determines whether to continue video shooting. If video shooting is to continue, the process proceeds to step S302.

[0067] In step S310, the system control unit 50 converts the image data into a moving image format to generate a moving image file, and records the moving image file with subject information, eye information, and user state information added as attribute information.

[0068] Next, a method for acquiring eyeball information using the event sensor 163 will be described with reference to FIG. 3B. FIG. 3B shows a subroutine for acquiring eyeball information performed in step S307 of FIG. 3A, and illustrates an example in which the accumulation period is a division of the video frame rate. The accumulation period of event data being a division of the video frame rate means that the accumulation time of event data is an integer fraction of one frame period. Here, the accumulation time of event data is assumed to be 1 / DR (DR is an integer) of one frame time. The integer DR corresponds to the division ratio.

[0069] In step S311, the event data calculation unit 164 receives the display synchronization data signal transmitted by the system control unit 50 in step S304.

[0070] In step S312, the system control unit 50 causes the event data calculation unit 164 to start accumulating the event data output from the event sensor 163.

[0071] In step S313, the system control unit 50 determines whether the time elapsed since the start of event data accumulation (S312) has reached the predetermined accumulation time (1 frame period / DR). If the determination result is YES, the system control unit 50 proceeds to step S314, and if the determination result is NO, the system control unit 50 continues accumulating the event data.

[0072] In step S314, the system control unit 50 causes the event data calculation unit 164 to end the accumulation of event data.

[0073] In step S315, the system control unit 50 calculates the event data and obtains first eye information in the event data calculation unit 164. The event data calculation unit 164 performs calculations on the event data accumulated during steps S312 to S314 to obtain the first eye information.

[0074] In step S316, the system control unit 50 associates the first eyeball information 1 with the frame number included in the display synchronization data signal and stores them in the memory 32.

[0075] In step S317, the system control unit 50 determines whether the product of the elapsed time since the synchronization data acquisition (S311) and the division ratio DR reaches the frame period. If the determination result is YES, the system control unit 50 proceeds to step S318, and if the determination result is NO, the system control unit 50 returns to step S312 and starts re-accumulation.

[0076] In step S318, the system control unit 50 interprets the results of the first eyeball information 1 obtained the number of times corresponding to the frequency division ratio DR, and obtains new second eyeball information.

[0077] In step S319, the system control unit 50 stores the second eyeball information 2 and the frame number included in the display synchronization data signal in the memory 32 in association with each other.

[0078] Here, for example, the first eye information is the occurrence or non-occurrence and amplitude of a microsaccade, which is a parameter that serves as an index of the gaze range. By setting a relatively short accumulation time, it becomes possible to identify rapid eye movements such as microsaccades. If the number is set to 3, it is possible to obtain the occurrence and amplitude of three microsaccades during one frame period. The second eye information is, for example, the frequency of occurrence of microsaccades, which is a parameter that serves as an index of the degree of concentration.

[0079] Next, a method for acquiring eyeball information using the event sensor 163 will be described with reference to Fig. 3C. Fig. 3C shows a subroutine for acquiring eyeball information performed in step S307 of Fig. 3A, and illustrates an example in which the accumulation period is a multiple of the video frame rate. The event data accumulation period being a multiple of the video frame rate means that the accumulation time of the event data is an integer multiple of one frame period. Here, the accumulation time of the event data is MR times one frame period (MR is an integer). The integer MR corresponds to the multiplication ratio.

[0080] Steps S321 and S322 are similar to steps S311 and S312, and therefore a description thereof will be omitted.

[0081] In step S323, the system control unit 50 determines whether the time elapsed since the start of event data accumulation (S322) has reached the predetermined accumulation time (1 frame period × MR). If the determination result is YES, the system control unit 50 proceeds to step S324, and if the determination result is NO, the system control unit 50 continues to accumulate the event data.

[0082] Steps S324 to S326 are the same as steps S314 to S316, and therefore a description thereof will be omitted. Note that the eyeball information obtained in step S325 is referred to as third eyeball information.

[0083] Here, for example, the eyeball information 3 is the amount of relative positional deviation between the eyeball (eye) 160 and the eyeball detection unit 161. By setting a relatively long accumulation time, it is possible to detect eyeball information that rarely occurs, such as whether or not positional deviation of the eyelid or caruncle has occurred during the accumulation time. The detected amount of positional deviation can be used, for example, to calibrate the gaze position detection result or as one of the indices of concentration level.

[0084] 3B and 3C are merely examples, and a subroutine may be provided for each type of eyeball information to be detected. For example, acquisition of eyeball information using a first division ratio (e.g., 1 / 3) and acquisition of eyeball information using a second division ratio (e.g., 1 / 2) may be performed in parallel. Similarly, acquisition of eyeball information using a first multiplication ratio (e.g., 2x) and acquisition of eyeball information using a second multiplication ratio (e.g., 3x) may be performed in parallel. On the other hand, if different types of eyeball information can be detected in the same accumulation time, a common subroutine may be used.

[0085] 4 is a time chart showing the concept of the information linking method in the first embodiment of the present invention. Image data 1001 represents captured image data captured by the imaging unit 22 at a predetermined frame rate and displayed on the EVF 29. Event data 1002 represents event data acquired by the event sensor 163. Here, at time t k From t k+1Event data that occurs between time t k This relates to changes in the photographer's eye movement resulting from visually viewing the image data 1001 displayed on the EVF 29.

[0086] The accumulation time 1003 represents the time from when accumulation starts to when it ends in the event data calculation unit 164. Accumulation starts when the display synchronization data signal is received in step S304, and ends when the predetermined division ratio DR is reached. For example, the accumulation time 1003 is the display frame rate (T=t k+1 -t k ) is a 3-division ratio, and the accumulation start time is time t k The event data calculation unit 164 interprets the event data accumulated during the accumulation time 1003 and outputs eyeball information 1004.

[0087] The accumulation time 1005 is, for example, a 2-division ratio of the display frame rate. The calculation unit 164 interprets the event data accumulated during the accumulation time 1005 and calculates eyeball information. Outputs 1006.

[0088] On the other hand, the accumulation time 1007 is twice the display frame rate. The event data calculation unit 164 interprets the event data accumulated during the accumulation time 1007 and outputs eyeball information 1008.

[0089] For example, user state information 1009 is user state information that is output as a result of inputting eyeball information 1004 and eyeball information 1006. Furthermore, user state information 1010 may be user state information that is output as a result of inputting subject information obtained from image data 1001 and eyeball information 1008.

[0090] In the above configuration, the accumulation time is set to the division ratio or multiplication ratio of the display frame rate. By setting the accumulation time as the division ratio, the next image data display time t k+1In this configuration, the image data display time and the event data accumulation start time are synchronized. If the accumulation time is a multiplication ratio, the image data display time and the event data accumulation start time are synchronized at the image data display time after the multiplication ratio number of steps. Since synchronized eyeball information is used as input to obtain user state information, the user state information (e.g., gaze level) is also information synchronized with the image data display time, just like the eyeball information. This configuration makes it easy to obtain user state information (e.g., subject preference) obtained from the correlation between subject information and eyeball information.

[0091] In the examples of Figures 3 and 4, the accumulation time is a predetermined fixed value, but it may also be configured so that a predetermined number of event data is used as a threshold and event data is accumulated until the number exceeds the threshold. A variable accumulation time is useful for acquiring eye information whose occurrence timing is unknown (for example, whether or not a blink occurred). However, even in a configuration with a variable accumulation time, the accumulation time is set to a division ratio or multiplication ratio of the display frame rate. In other words, the accumulation of event data ends when the number of event data acquired after the accumulation start time exceeds a predetermined threshold and the elapsed time coincides with an integer division of the frame period or an integer multiple of that time. This makes it easy to synchronize the image data display time with the event data accumulation start time.

[0092] Fig. 5 is a diagram showing an example of the file structure of video data to which attribute information has been assigned by the processing of Fig. 3. File 1101 shows the entire video file recorded by digital camera 100. Data 1102 shows video image data captured by digital camera 100. Data 1103 shows attribute information assigned to event data 1002. Data 1104 shows attribute information recorded by the processing of Fig. 3. The attribute information is metadata such as subject information, eye information, and user state information, and each piece of information is associated with the frame number included in the display synchronization data signal.

[0093] 5 is an example of the attribute information, and is not limited to this. Quantitative information such as a microsaccade amplitude value may also be used as the attribute information.

[0094] In the configuration of this embodiment, each piece of information is associated and expressed by the frame number included in the display synchronization data signal, but the display synchronization data signal may be configured to include display synchronization time information. In that case, 1104 is configured to be associated by the display time information.

[0095] Furthermore, in the configuration of this embodiment, the attribute information is assigned to the moving image data, but it is of course also possible to assign it to the still image data.

[0096] In addition, in the configuration of this embodiment, the photographer views the image data displayed on the EVF 29, but the configuration is not limited to this. The present invention can also be applied to a configuration of a SLR camera or a see-through type head mounted display equipped with an imaging means, in which case the accumulation start time of the event data is synchronized with the imaging frame rate.

[0097] Furthermore, while this embodiment is an example in which the present invention is applied to a digital camera 100, the present invention can be applied to any electronic device configured to store information by associating frame-based data with event-based data. For example, the imaging unit 22 is not an essential component, and step S301 may be the start of playback of moving image data. Playback display is started, in which the moving image data stored on the recording medium 200 is played back and displayed on the EVF 29. Step S302 is for acquiring moving image data. There are no changes from steps S303 to S308. Step S309 determines whether to end playback display. If YES, the process proceeds to step S3110. In step S310, attribute information is assigned to the display data. With the above configuration, the present invention can also be applied to display devices, such as head-mounted displays.

[0098] Example 2 A digital camera 100 according to a second embodiment of the present invention will be described below with reference to Figures 6A to 6F and 7. In this embodiment, the digital camera 100 records a captured video by linking the user's state regarding the line of sight with work process information.

[0099] 6A to 6F are diagrams for explaining, in chronological order, the relationship between the image captured by digital camera 100 and displayed on EVF 29, and the user's state and work process information related to the line of sight. Common parts in each diagram are expressed by adding the corresponding letter a to f to the end of the figure number, and explanations of the common parts will be omitted.

[0100] 6A shows the display image of the EVF 29 that the user is looking at, and associated information, at time t0 when video shooting started. A subject 600a is captured by the imaging unit 22, and its size and position within the image data are identified before it is displayed on the EVF 29. A point 601 indicates the user's gaze position at time t0. Information 602 shows an example of work process information that is recorded in association with the user's state. In this embodiment, the work process information indicates the shooting mode, the state of the camera, and the elapsed time since gaze detection started.

[0101] FIG. 6B is a diagram showing a predetermined time Δt after time t0 (FIG. 6A). Arrow 603 indicates the trajectory of the user's microsaccade amplitude (movement of gaze position) during the predetermined time Δt. Area 604 is a circular area including arrow 603 and is defined as the gaze range. In the example of FIG. 6B, it can be seen that the user is holding digital camera 100 in his / her hand, is shooting in video mode while stationary, and is gazing at the upper left corner of the angle of view of the EVF 29. The predetermined time Δt can be set to any time and may be shorter than the time between display frames of the EVF 29.

[0102] FIG. 6C is a diagram of time t1, after a predetermined time Δt or more has elapsed since time t0 (FIG. 6A). Area group 605 shows the gaze ranges for each predetermined time Δt. Arrow 606 is a line connecting the centers of the circular areas included in area group 605, and is defined as the direction in which the user's gaze moves. In the example of FIG. 6C, it can be seen that the user is moving their gaze from the upper left of the angle of view of the EVF 29 to the face of subject 600c.

[0103] 6D is a diagram of time t2, a predetermined time after time t1 (FIG. 6C). Region group 607 shows the gaze range from time t1 to time t2. In the example of FIG. 6D, it can be seen that the user is gazing at the entire face of subject 600d while taking the photograph.

[0104] FIG. 6E is a diagram of time t3, a predetermined time after time t2 (FIG. 6D). Area group 608 shows the gaze range from time t2 to time t3. Furthermore, work process information 602e indicates that the user has transitioned from handheld (still) shooting to handheld (walking) shooting. In the example of FIG. 6E, it can be seen that the user is walking closer to subject 600e while gazing at the entire face of subject 600e.

[0105] FIG. 6F is a diagram of time t4, a predetermined time after time t3 (FIG. 6E). Region group 609 shows the gaze range from time t3 to time t4. Arrow 610, a line connecting the centers of each circular region included in region group 609, indicates that the user's gaze moved from the right to the left on the EVF 29. Furthermore, work process information 602f indicates that the user transitioned from handheld (walking) shooting to handheld (still) shooting. Double circles 611 and 612 indicate that the user's gaze position was detected again after it could not be detected for a predetermined period of time. Double circle 611 indicates the gaze position just before it became undetectable, and double circle 612 indicates the gaze position just after it was detected again. The period from double circle 611 to double circle 612 can be defined as the period during which the user blinked. In the example of FIG. 6F, it can be seen that subject 600f and the user's gaze range moved from the right to the left on the EVF 29. Furthermore, it can be seen that the user blinks once when moving their gaze from right to left on the EVF29.

[0106] 7 is a flowchart for linking and recording user status and work process information with an image displayed on the EVF 29 of the digital camera 100. This flow starts when the eyepiece unit 16 detects that the user's eyeball (eye) 160 is approaching.

[0107] Steps S701 to S706 are the same as steps S301 to S307 in FIG. 3, so a description thereof will be omitted.

[0108] In step S708, the system control unit 50 inputs the subject information and eyeball information to the user state determination unit 165 and acquires various pieces of user state information as output results. At this time, the information input to the user state determination unit 165 is the subject information and eyeball information associated with the same associated frame number. The system control unit 50 associates the acquired user state information with the frame number at the time of information acquisition and stores it in the memory 32. One example of user state information is the gaze range, which can be calculated from the microsaccade information acquired in step S708. The gaze range can be defined, for example, as a circular area that includes the trajectory of the microsaccade amplitude (movement of gaze position) over a predetermined time Δt, but the calculation method is not limited to this.

[0109] In step S709, the system control unit 50 acquires the shooting mode and the state of the digital camera 100 obtained from the attitude detection unit 55 as user work process information. Examples of shooting modes include video shooting mode, still image shooting mode, and still image shooting preparation mode. Examples of the state of the digital camera 100 include tripod shooting state, handheld shooting state, walking shooting state (walking shooting), running shooting state (running shooting), and panning shooting state. The system control unit 50 can determine these states based on the output of the gyro sensor and acceleration sensor included in the attitude detection unit 55. The system control unit 50 records the acquired work process information in memory 32, linking it to the frame number at the time each piece of information was acquired.

[0110] Steps S710 and S711 are the same as steps S309 and S310 in FIG. 3, respectively, and therefore will be omitted.

[0111] Although the above explanations of FIGS. 6A to 6F and FIG. 7 are merely examples, by linking and recording the user's gaze-related state and work process information in chronological order, it is possible to obtain a more detailed picture of the user's intention to take a photograph. Furthermore, if the user is an expert, this recorded information can be expected to help beginners understand the expert's intentions in detail and lead to improved proficiency.

[0112] Example 3 A digital camera 100 according to a third embodiment of the present invention will be described below with reference to Figures 8 and 9. In this embodiment, the digital camera 100 records a captured still image by linking the user's state regarding the line of sight and work process information.

[0113] FIG. 8 is a diagram for explaining the relationship between the user state and work process information related to the line of sight, which are recorded in association with a still image.

[0114] An arrow 800 indicates the time axis, with time passing further to the right. Time 801 indicates the timing on the time axis when the first shutter switch 62 (hereinafter referred to as SW1) is pressed. Pressing SW1 starts various shooting preparation operations such as AF processing and AE processing. Time 802 indicates the timing on the time axis when the second shutter switch 63 (hereinafter referred to as SW2) is pressed. Pressing SW2 starts a series of shooting processing operations from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file.

[0115] Frame 803 is the first frame acquired by the imaging unit 22 after time 801. Frame 804 is the last frame acquired by the imaging unit 22 before time 802. Frame group 805 is frames acquired by the imaging unit 22 between frame 803 and frame 804, and the acquisition interval is determined by the frame rate of the imaging unit 22. Arrow 806 indicates the exposure time TV for the imaging unit 22, which is controlled by the system control unit 50. Captured image 807 is a still image acquired by the imaging unit 22 by exposure between time 802 and arrow 806.

[0116] An arrow 808 indicates a period TSW1 from time 801 to acquisition of a frame 804. During this period, the system control unit 50 determines that the operation mode of the digital camera 100 is the still image capture mode SW1 (still image capture preparation operation in progress).

[0117] Arrow 809 indicates a predetermined period TSW2 that includes time 802. In FIG. 8, period TSW2 is expressed as the period from when frame 804 is acquired to when captured image 807 is acquired, but this is not limited to this and can be set arbitrarily. During this period, system control unit 50 determines that the operation mode of digital camera 100 is still image capture mode SW2 (still image capture in progress). Here, these operation modes are defined as work process information. However, this is not limited to these.

[0118] Data 810 indicates eyeball information and user status acquired during period TSW1 (during shooting preparation operation). Data 811 indicates eyeball information and user status acquired during period TSW2 (during shooting operation). Eyeball information here refers to at least one of gaze position, gaze movement direction, blink, and microsaccade information, and user status refers to the gaze range calculated from the eyeball information. However, eyeball information and user status are not limited to these.

[0119] 9 is a flowchart for linking and recording the user's state regarding the line of sight and work process information for a still image captured by digital camera 100. This flow starts when eyepiece unit 16 detects that the user's eyeball (eye) 160 is approaching.

[0120] In step S901, the user starts preparations for still image capture. Here, the timing when SW1 is pressed is considered to be the start of preparations for still image capture. At this time, the system control unit 50 transitions the operating mode of the digital camera 100 to the still image capture mode SW1 state. In still image capture mode SW1, images captured by the imaging unit 22 are displayed on the EVF 29 at a predetermined frame interval, just like when capturing moving images.

[0121] Steps S902 to S905 are the same as steps S302 to S305 in FIG. 3, respectively, and therefore will be omitted.

[0122] In step S906, the system control unit 50 analyzes the image data in the subject identification unit 70 and extracts subject information such as subject type, size, position, etc. The system control unit 50 stores the subject information in the memory 32 in association with the time when the information was acquired.

[0123] In step S907, the system control unit 50 calculates the event data obtained from the event sensor 163 using the event data calculation unit 164, and acquires eye information such as gaze position information, saccade information, microsaccade information, blink information, etc. The system control unit 50 stores the acquired eye information in the memory 32, linking each piece of information to the time when it was acquired.

[0124] In step S908, the system control unit 50 inputs the subject information and eyeball information to the user state determination unit 165 and acquires various pieces of user state information as output results. At this time, the system control unit 50 stores the acquired user state information in the memory 32 in association with the time when the information was acquired.

[0125] In step S909, the system control unit 50 detects the shooting mode and the posture from the posture detection unit 55. The state of the digital camera 100 obtained from the above is acquired as user work process information. The shooting mode here refers to the still image shooting mode SW1. The state of the digital camera 100 refers to tripod shooting state, handheld shooting state, walking shooting state, running shooting state, panning shooting state, etc., which can be determined by the system control unit 50 based on the output of the gyro sensor and acceleration sensor included in the attitude detection unit 55. The system control unit 50 records the acquired work process information in memory 32, linking it to the time when each piece of information was acquired.

[0126] In step S910, it is determined whether SW2 has been pressed. If it has been pressed, the process proceeds to step S911, and if not, the process proceeds to step S902.

[0127] In step S911, exposure of the imaging unit 22 is started under the control of the system control unit 50. At this time, the operation mode of the digital camera 100 is transitioned to the still image shooting mode SW2 state by the system control unit 50. However, the system control unit 50 recognizes the operation mode of the digital camera 100 as the still image shooting mode SW2 from the time when the frame 804 immediately before SW2 was pressed was acquired.

[0128] Steps S912 and S913 are the same as steps S907 and S908, respectively, and therefore will not be described here.

[0129] In step S914, the system control unit 50 acquires the shooting mode and the state of the digital camera 100 obtained from the attitude detection unit 55 as user work process information. The shooting mode here refers to the still image shooting mode SW2. The state of the digital camera 100 refers to tripod shooting state, handheld shooting state, walking shooting state, running shooting state, panning shooting state, etc., which the system control unit 50 can determine based on the output of the gyro sensor and acceleration sensor included in the attitude detection unit 55. The system control unit 50 then acquires the acquired state. The work process information is recorded in the memory 32 in association with the time when each piece of information was acquired.

[0130] In step S915, the various information obtained in steps S906 to S909 as well as the various information obtained in steps S912 to S914 are assigned as attribute information to the captured still image data and recorded.

[0131] 8 and 9 are merely examples, in this embodiment, the eyeball information, user state information, and work process information acquired during the shooting preparation operation and the shooting operation, respectively, are stored as attribute information in the captured still image data. In this way, by linking and recording the user state and work process information related to the line of sight chronologically, in addition to the effect of Example 2, it is possible to record the user's shooting intentions in more detail even when capturing still images. If the user is an expert, this recorded information is expected to help a beginner understand the expert's shooting intentions in detail and lead to improvement in proficiency.

[0132] <Other embodiments> Although the embodiments to which the present invention is applied have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0133] In the above example, image capture was used as an example of a task performed by a user, but the task content is not limited to image capture and may be any task. Examples of tasks other than image capture include manufacturing or processing goods, inspecting goods, operating equipment (such as driving a vehicle), and physical exercise.

[0134] Furthermore, the event sensor detects the eyeball and acquires eyeball information as event information, but the event sensor may detect something other than the eyeball.

[0135] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0136] 1000: Digital camera, 22: Image capture unit, 29: EVF, 32: Memory 50: System control unit, 161: Eyeball detection unit, 163: Event sensor 164: Event data calculation unit, 165: User state determination unit, 70: Object identification unit

Claims

1. frame data acquisition means for acquiring frame data generated at a predetermined frame period; an event data acquisition means for detecting a change in luminance of light incident on a pixel and continuously acquiring event data, which is information about a pixel in which a luminance change has occurred, asynchronously with other pixels; an event data calculation means for synchronizing an accumulation start time for starting accumulation of the event data with a period of the frame data, setting an accumulation time from the accumulation start time to an accumulation end time as a time that is an integer fraction or an integer multiple of the frame period, and extracting event information from the event data accumulated during the accumulation time; a recording means for recording the event information in association with the frame data synchronized with the accumulation start time; Equipped with the frame data acquisition means is an imaging means, the event data acquisition means acquires the event data from the eyes of a user looking at a display device on which the frame data is displayed; The event data calculation means calculates eyeball information related to the user's eyes from the event data. An electronic device characterized by:

2. A frame data acquisition means for acquiring frame data generated at a predetermined frame period; an event data acquisition means for detecting a change in luminance of light incident on a pixel and continuously acquiring event data, which is information about a pixel in which a luminance change has occurred, asynchronously with other pixels; an event data calculation means for synchronizing an accumulation start time for starting accumulation of the event data with a period of the frame data, setting an accumulation time from the accumulation start time to an accumulation end time as a time that is an integer fraction or an integer multiple of the frame period, and extracting event information from the event data accumulated during the accumulation time; a recording means for recording the event information in association with the frame data synchronized with the accumulation start time; Equipped with the frame data acquisition means is an imaging means for imaging a field of view of a user, the event data acquisition means acquires the event data from the user's eyes; The event data calculation means calculates eyeball information related to the user's eyes from the event data. An electronic device characterized by:

3. The eyeball information is Gaze position information regarding the user's gaze position; saccadic information regarding the direction or speed of the saccade; microsaccade information regarding the frequency or amplitude of microsaccades; Pupil information relating to pupil size or its change amount; Blink information about the rate or frequency of blinks including at least one of 3. The electronic device according to claim 1 or 2.

4. further comprising a user state determination means for acquiring user state information relating to the state of the user from the eyeball information, the recording means records the user status information in association with the frame data synchronized with the accumulation start time.

4. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

5. The user state information includes at least one of the user's gaze range, gaze level, bird's-eye view level, concentration level, fatigue level, and preference level.

5. The electronic device according to claim 4.

6. further comprising a subject identification means for acquiring subject information relating to the subject included in the frame data; the recording means records the subject information in association with the frame data synchronized with the accumulation start time.

6. The electronic device according to claim 4 or 5.

7. the subject information includes at least one of the type of subject included in the frame data, size information within the frame data, and position information; 7. The electronic device according to claim 6.

8. the user state determination means determines the user state information from the eyeball information and the subject information.

8. The electronic device according to claim 6 or 7.

9. the event data calculation means terminates the accumulation of the event data when the number of the event data acquired after the accumulation start time exceeds a specified threshold and coincides with an integer multiple of a time that is an integer fraction of the frame period or an integer multiple of the frame period; 9. An electronic device according to claim 1.

10. An electronic device according to any one of claims 1 to 9; Photography means, An imaging device comprising:

11. 1. A method for controlling an electronic device, comprising: A frame data acquisition step for acquiring frame data generated at a predetermined frame period. Pu and, an event data acquisition step of detecting a change in luminance of light incident on a pixel and continuously acquiring event data, which is information about a pixel that has experienced a luminance change, asynchronously with respect to other pixels; an event data calculation step of synchronizing an accumulation start time at which accumulation of the event data is started with a period of the frame data, setting an accumulation time from the accumulation start time to an accumulation end time as a time that is an integer fraction or an integer multiple of the frame period, and extracting event information from the event data accumulated during the accumulation time; a recording step of recording the event information in association with the frame data synchronized with the accumulation start time; Including, the frame data acquisition step acquires the frame data from an imaging means; In the event data acquisition step, the event data is acquired from the eyes of a user who is viewing a display device on which the frame data is displayed; In the event data calculation step, eyeball information relating to the user's eyes is calculated from the event data.

10. A method for controlling an electronic device, comprising:

12. On the computer, a frame data acquisition step of acquiring frame data generated at a predetermined frame period; an event data acquisition step of detecting a change in luminance of light incident on a pixel and continuously acquiring event data, which is information about a pixel that has experienced a luminance change, asynchronously with respect to other pixels; an event data calculation step of synchronizing an accumulation start time at which accumulation of the event data is started with a period of the frame data, setting an accumulation time from the accumulation start time to an accumulation end time as a time that is an integer fraction or an integer multiple of the frame period, and extracting event information from the event data accumulated during the accumulation time; a recording step of recording the event information in association with the frame data synchronized with the accumulation start time; Execute the frame data acquisition step acquires the frame data from an imaging means; In the event data acquisition step, the event data is acquired from the eyes of a user who is viewing a display device on which the frame data is displayed; In the event data calculation step, eyeball information relating to the user's eyes is calculated from the event data. program.

Images