Electronic device, electronic device control method, program, and recording medium

Abstract

To acquire highlight on the basis of the state of a user.SOLUTION: An electronic apparatus comprises: acquisition means for acquiring user's eyeball information; and recording means for recording an image corresponding to a visual field of the user when the state of the user satisfies a predetermined condition, on the basis of the eyeball information acquired by the acquisition means.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a control method for an electronic device, a program, and a recording medium. [Background technology]

[0002] When a user is using an electronic device such as a camera, smart glasses, or head-mounted display and is concentrating on viewing the scenery or content, it is difficult for the user to intentionally specify the timing (recording timing) to record (acquire) highlights. Therefore, it is preferable that the recording timing be determined automatically.

[0003] As a method for automatically determining the recording timing, for example, Patent Document 1 discloses a method for determining the recording timing based on the user's sense of tension, and Patent Document 2 discloses a method for determining the recording timing based on the movement of an electronic device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-113609 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-17598 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the methods of Patent Document 1 and Patent Document 2, the timing of recording is determined based on the user's sense of tension or the movement of the electronic device, so even if the user wants to record highlights, the highlights may not be recorded.

[0006] Therefore, an object of the present invention is to record highlights based on the user's state. [Means for solving the problem]

[0007] A first aspect of the present invention is 1. An electronic device comprising: an acquisition means for acquiring eyeball information of a user; an estimation means for estimating a size of a gaze range of the user based on the eyeball information acquired by the acquisition means, and estimating whether or not the user is tracking a specific object with his or her eyes based on the eyeball information and the size of the gaze range; and a recording means for recording an image corresponding to the user's field of view when the estimation means estimates that the user is tracking a specific object with his or her eyes, wherein the estimation means estimates that the user is tracking a specific object with his or her eyes when the size of the gaze range is equal to or smaller than a third threshold and a saccade speed is equal to or larger than a fourth threshold. is. A second aspect of the present invention is an electronic device comprising: an acquisition means for acquiring eye information of a user; an estimation means for estimating the size of the user's gaze range based on the eye information acquired by the acquisition means and estimating whether the user is in a state of looking down based on the size of the gaze range; and a recording means for recording an image corresponding to the user's field of view when the estimation means estimates that the user is in a state of looking down, wherein the estimation means estimates that the user is in a state of looking down when the size of the gaze range is larger than a fifth threshold for a predetermined time.

[0008] The present invention 3 The embodiment of a recording step of recording an image corresponding to the user's field of view when it is estimated in the estimation step that the user is tracking a specific object with his / her eyes, wherein the estimation step estimates that the user is tracking a specific object with his / her eyes when the size of the gaze range is equal to or smaller than a third threshold and the speed of a saccade is equal to or larger than a fourth threshold; is. A fourth aspect of the present invention is a control method for an electronic device, comprising: an acquisition step of acquiring eye information of a user; an estimation step of estimating the size of the user's gaze range based on the eye information acquired in the acquisition step and estimating whether the user is in a state of looking down based on the size of the gaze range; and a recording step of recording an image corresponding to the user's field of view when it is estimated in the estimation step that the user is in a state of looking down, wherein in the estimation step, it is estimated that the user is in a state of looking down if the size of the gaze range is larger than a fifth threshold for a predetermined time.

[0009] The present invention 5 The third aspect is a program for causing a computer to execute each step of the above-described method for controlling an electronic device.

[0010] The present invention 6 The third aspect is a computer-readable recording medium storing a program for causing a computer to execute each step of the above-described method for controlling an electronic device. [Effects of the Invention]

[0011] According to the present invention, highlights can be recorded based on the user's state. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is an external view of a camera according to a first embodiment; [Figure 2] Block diagram of a camera according to a first embodiment [Figure 3] Block diagram of smart glasses according to Example 1 [Figure 4] Flowchart of highlight recording flag setting process according to the first and second embodiments [Figure 5] Flowchart of recording process of highlight still image according to the first embodiment [Figure 6] 1 is a flowchart of a highlight video recording process according to the first embodiment; [Figure 7] FIG. 1 is a diagram showing a highlight video according to the first embodiment. [Figure 8] Block diagram of a head-mounted display according to a second embodiment. [Figure 9] Flowchart of recording process of highlight still image according to the second embodiment [Figure 10] 10 is a flowchart of a highlight video recording process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Example 1 Hereinafter, a first embodiment of the present invention will be described with reference to the accompanying drawings. As the first embodiment, an example in which the present invention is applied to a camera or smart glasses will be described.

[0014] <Configuration explanation> 1(a) to 1(c) are external views of a camera 100 (digital camera) as an example of an electronic device to which the present invention can be applied. FIG. 1(a) is a top view, FIG. 1(b) is a perspective view, and FIG. 1(c) is a rear view. The EVF 101 is an eyepiece-type EVF (Electronic Viewfinder). The EVF 101 displays captured images and various information, including a live view during shooting, various information, and a GUI (Graphical User Interface) for a setting screen. The photographing lens unit 102 includes a zoom lens and a focus lens protected by a barrier member, and actuators that drive them. The photographing lens unit 102 optically controls the photographing angle of view in response to user operations. The recording medium 103 is a recording medium such as a memory card or a hard disk. The recording medium slot 110 is a slot for storing the recording medium 103. When the recording medium 103 is stored in the recording medium slot 110, it becomes possible for the recording medium 103 to communicate with the camera 100. The lid 104 is a lid for the recording medium slot 110.

[0015] The power switch 105 is an operation unit that switches the power on / off of the camera 100. The MENU button 106 is an operation unit that switches the menu display between the ON state and the OFF state. The shutter button 107 is an operation unit that issues shooting instructions. The shutter button 107 is an operation unit that also serves as a next button that issues instructions for the next operation. The shutter button 107 functions as a shutter button when the menu display is in the OFF state (live view display state), and functions as a next button when the menu display is in the ON state. The Zoom tele button 108 is an operation unit that issues a zoom instruction to the telephoto side, and the Zoom wide button 109 is an operation unit that issues a zoom instruction to the wide side (wide-angle side).

[0016] 2 is a block diagram showing an example configuration of the camera 100. The photographing lens unit 102 includes a first fixed lens group 219, a zoom lens group 220, an aperture 221, a third fixed lens group 222, and a focus lens group 223. The zoom lens group 220 is driven by a DC motor (zoom motor) serving as an actuator, and the amount of drive thereof is measured by an encoder. The aperture group 221 is driven by an actuator, and the amount of drive thereof is measured by an encoder. The focus lens group 223 has both a focus function and a function as a compensator lens that corrects movement of the focal plane accompanying drive of the zoom lens group 220. The focus lens group 223 is also driven by a pulse motor (focus motor) serving as an actuator, and the amount of drive thereof is measured by an encoder.

[0017] The shutter 201 controls the exposure time of the image capturing unit 204 under the control of the system control unit 208. The barrier 202 is a barrier member that prevents the image capturing system, including the photographing lens unit 102, the shutter 201, and the image capturing unit 204, from becoming dirty or being damaged.

[0018] The imaging unit 204 is an imaging element (image sensor) composed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The A / D converter 205 is used to convert an analog signal output from the imaging unit 204 into a digital signal. While the event sensor 163 described below is an event-based vision sensor (an asynchronous event-based sensor), the imaging unit 204 is a synchronous frame-based sensor.

[0019] The image processing unit 206 performs predetermined processing (pixel interpolation, resizing such as enlargement or reduction, color conversion, etc.) on data from the A / D converter 205 or data from the memory control unit 207. The image processing unit 206 also performs predetermined arithmetic processing using captured image data. The system control unit 208 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 206. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing to be performed. The image processing unit 206 also performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the arithmetic results obtained.

[0020] The memory control unit 207 controls the transmission and reception of data between the A / D converter 205, the image processing unit 206, and the memory 209. The output data from the A / D converter 205 is written into the memory 209 via the image processing unit 206 and the memory control unit 207, or via the memory control unit 207 without via the image processing unit 206.

[0021] The system control unit 208 is a control unit made up of at least one processor or circuit, and controls the entire camera 100. The system control unit 208 realizes each process described below by executing a program recorded in a nonvolatile memory 211. The system control unit 208 performs display control by controlling the memory 209, D / A converter 210, EVF 101, etc.

[0022] The memory 209 stores image data obtained by the imaging unit 204 and converted into digital data by the A / D converter 205, as well as image data to be displayed on the EVF 101. The memory 209 has a storage capacity sufficient to store a predetermined number of still images, a predetermined amount of moving images, and audio. The memory 209 also serves as a memory for displaying images (video memory).

[0023] The D / A converter 210 converts image display data stored in the memory 209 into an analog signal and supplies it to the EVF 101. The display image data written to the memory 209 is displayed by the EVF 101 via the D / A converter 210. The EVF 101 is a display such as an LCD, and performs display according to the analog signal from the D / A converter 210. Digital signals that have been A / D converted by the A / D converter 205 and stored in the memory 209 are converted to analog by the D / A converter 210 and sequentially transferred to and displayed on the EVF 101, thereby enabling live view display (through image display).

[0024] The nonvolatile memory 211 is an electrically erasable and recordable memory, and may be, for example, an EEPROM. The nonvolatile memory 211 stores constants, programs, etc. for the operation of the system control unit 208. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0025] The system memory 212 is, for example, a RAM, and the system control unit 208 loads constants and variables for the operation of the system control unit 208, programs read from the nonvolatile memory 211, and the like into the system memory 212. The system timer 213 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0026] The operation unit 203 is an input unit that accepts operations from the user (user operations) and is used to input various operational instructions to the system control unit 208. The operation unit 203 includes a shutter button 107 and other operation members 216. The other operation members 216 include a power switch 105, a MENU button 106, a Zoom Tele button 108, and a Zoom Wide button 109. When the system control unit 208 is notified that the MENU button 106 has been pressed, it transitions to a menu display ON state if in a live view display state, and displays a menu screen on the EVF 101 that allows various settings to be made. The user can intuitively make various settings, such as by specifying a selection item using the menu screen displayed on the EVF 101 and the operation unit 203. Furthermore, when the system control unit 208 is notified that the MENU button 106 has been pressed while the menu display is ON, it controls the system to return to a live view display state.

[0027] The shutter button 107 includes a first shutter switch 214 and a second shutter switch 215. The first shutter switch 214 is turned on when the shutter button 107 provided on the camera 100 is pressed halfway (a shooting preparation instruction) during operation, and generates a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 208 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0028] The second shutter switch 215 is turned on when the shutter button 107 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 208 starts a series of photographing processing operations based on the second shutter switch signal SW2, from reading out a signal from the imaging unit 204 to writing image data of the captured image to the recording medium 103 as an image file.

[0029] The power supply control unit 217 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the battery type, and the remaining battery power. The power supply control unit 217 also controls the DC-DC converter based on the detection results and instructions from the system control unit 208, and supplies the required voltage for the required period to each unit, including the recording medium 103. The power supply unit 218 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or Li batteries, an AC adapter, etc.

[0030] The network interface 224 communicates with a network 225 such as a LAN (Local Area Network) or the Internet based on instructions from the system control unit 208. When communicating, the system control unit 208 can transmit display image data written in the memory 209 to an external display device via the network interface 224. It can also receive information via the network interface 224.

[0031] The recording medium interface 226 is a communication interface with the recording medium 103 such as a memory card or a hard disk. The recording medium 103 is a recording medium for recording captured images, and is configured from a semiconductor memory, a magnetic disk, or the like.

[0032] The object identification unit 227 analyzes the image data obtained by the imaging unit 204 and identifies the type of object. The object identification unit 227 performs the above processing by using, for example, a convolutional neural network that is widely used in image recognition.

[0033] The eyeball detection unit 161 is composed of an eyeball detection lens 162, an event sensor 163, and an event data calculation unit 164, and is capable of acquiring eyeball information relating to the state of the eye 160 of the user looking through the viewfinder.

[0034] The infrared light emitted from the infrared light emitting diode 228 is reflected by the user's eyeball, and the reflected infrared light passes through the eyeball detection lens 162 and forms an image on the imaging surface of the event sensor 163 .

[0035] The event sensor 163 is an event-based vision sensor that detects changes in the luminance of light incident on each pixel and outputs information about pixels where the luminance change occurred asynchronously with other pixels. The data output from the event sensor 163 includes, for example, the position coordinates of the pixel where the luminance change (event) occurred, the polarity (positive or negative) of the luminance change, and timing information corresponding to the time the event occurred. This data will be referred to as event data hereafter. Compared to synchronous frame-based sensors such as the existing imaging unit 204, the event sensor 163 eliminates redundancy in the output information and offers high-speed operation, a high dynamic range, and low power consumption. However, because the event data (information about pixels where the luminance change occurred) is output asynchronously with other pixels, special processing is required to determine the correlation between the event data. To determine the correlation between the event data, it is necessary to accumulate the event data output from the event sensor 163 over a predetermined period of time and perform various arithmetic operations on the results.

[0036] The event data calculation unit 164 is a calculation unit for acquiring (detecting) eye information based on event data that is continuously and asynchronously output from the event sensor 163. For example, the event data calculation unit 164 accumulates event data that occurs over a predetermined period of time and processes the data as a set of data to acquire eye information. By changing the accumulation time for accumulating the event data, it is possible to acquire a plurality of pieces of eye information that occur at different speeds.

[0037] The eyeball information includes, for example, gaze position information regarding gaze position (the position where the user is looking), saccade information regarding the direction and speed of saccades, and microsaccade information including the frequency and size (amplitude) of microsaccades. The eyeball information may also include information regarding eyeball movements other than saccades and microsaccades, pupil information regarding pupil size and changes therein, and blink information regarding the speed and number of blinks. These pieces of information are merely examples, and the eyeball information is not limited to these. The event data calculation unit 164 may map the event data for the accumulation time as one frame of image data based on event occurrence coordinates (the position coordinates of pixels where a luminance change (event) occurred) and perform image processing. With this configuration, eyeball information can be acquired from one frame of image data obtained by mapping the event data for the accumulation time using frame-based image processing.

[0038] The user state estimation unit 165 is an estimation unit that estimates the state of the user based on the eyeball information obtained by the event data calculation unit 164. For example, it is possible to estimate the size (width) of the gaze range or the degree of gaze (bird's-eye view degree) from the frequency and amplitude of microsaccades. Here, the gaze range is synonymous with the attention range or the range of attention. The gaze degree is an index that is higher the narrower the gaze range, and lower the wider it is. The bird's-eye view degree is defined as the antonym of the gaze degree. Furthermore, it is possible to estimate the user's concentration level (state of concentration) or fatigue level from the frequency and amplitude of microsaccades, the size and amount of change of the pupil, and the speed and number of blinks. Furthermore, the user's excitement level is related to the frequency and speed of microsaccades and pupil diameter, and can be estimated from these parameters. The excitement level is an index that is higher when the user is looking at an object of high preference (such as a favorite face) and lower when looking at an object of low preference, so it can be considered as a preference level. The user state estimation unit 165 may be configured by a neural network that receives, for example, parameters related to eyeball information and the identification result of the object identification unit 227 as input and outputs information related to the user's state (hereinafter referred to as user state information). However, the configuration of the user state estimation unit 165 is not limited to the above configuration. The eyeball information used by the user state estimation unit 165 and the estimation result of the user state estimation unit 165 are not limited to those described above.

[0039] The gaze input setting unit 166 sets whether the processing of the eyeball detection unit 161 is enabled or disabled via the system control unit 208. The gaze input setting unit 166 can also set parameters and conditions related to the processing of the event data calculation unit 164 and the user state estimation unit 165. For example, the user can arbitrarily make these settings from a menu screen or the like.

[0040] The system control unit 208 can also obtain information about the area of ​​the EVF 101 in which the subject (object) being photographed is displayed and at what size. Furthermore, the eyeball detection unit 161 can also obtain information about the area of ​​the EVF 101 at which the user is directing their gaze. This allows the system control unit 208 to determine which area of ​​the subject the user is looking at.

[0041] The timing detection unit 167 detects a recording timing, which is a timing for recording a highlight (an image (still image or video) of a highlight scene). The timing detection unit 167 detects the recording timing based on the eyeball information obtained by the event data calculation unit 164 or the user's state estimated by the user state estimation unit 165. For example, the timing detection unit 167 detects, as the recording timing, a timing when a predetermined condition is satisfied, such as when the value of the eyeball information or the user state information falls within a certain range or when it changes by a certain amount. The timing detection unit 167 may also take into account a time-related condition and detect, as the recording timing, a timing when the value of the eyeball information or the user state information falls within a certain range for a certain period of time or when it changes by a certain amount. The predetermined condition may be a combination of the eyeball information or the user state information. For example, the predetermined condition is satisfied when the user's excitement level exceeds a certain value, or when the size of the user's gaze range falls below a certain value and the saccade speed exceeds a certain value. The predetermined condition is set by the system control unit 208. The system control unit 208 may set conditions stored in the system memory 212 or may set conditions specified by the user. The predetermined conditions may include multiple conditions. However, the predetermined conditions are not limited to those described above. The eye information or user state information used to detect the recording timing is not limited to those described above.

[0042] FIG. 3 is a block diagram showing an example of the configuration of smart glasses 300. In the first embodiment, the present invention will be described as being applied to the camera 100, but may also be applied to eyeglass-type electronic devices (wearable electronic devices) such as the smart glasses 300. The smart glasses 300 capture an image of a range corresponding to the user's field of view and record the image corresponding to the user's field of view as a highlight. While the user looks at the EVF 101 in the camera 100 of FIG. 2, the user looks at the outside world through the lens 301 in the smart glasses 300 of FIG. 3. The focal length of the lens 301 varies for each user and is set to match the user's focus.

[0043] The present invention may also be applied to electronic telescopes (binoculars or monoculars), etc. For example, even if an electronic device does not have the storage capacity to record videos of several tens of minutes, it can record highlights according to the user's state, allowing it to record scenes that the user would like to rewatch.

[0044] <Highlight Recording Flag Setting Process> Next, a setting process for setting a highlight recording flag, which is a flag indicating the timing of highlight recording, will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the setting process. Each process in the flowchart of Fig. 4 is realized by the system control unit 208 loading a program stored in the nonvolatile memory 211 into the system memory 212, executing it, and controlling each functional block. For example, the setting process starts when the camera 100 is powered on.

[0045] In step S401, the system control unit 208 controls the event data calculation unit 164 to acquire eyeball information of the user viewing the image displayed on the EVF 101. In step S401, the eyeball information includes the frequency of occurrence of microsaccades, the amplitude (magnitude) of the microsaccades, and the speed of the saccades. Note that the eyeball information may also include other information such as pupil diameter. Note that when the user views the image displayed on the EVF 101, the range of the image displayed on the EVF 101, i.e., the range of the image captured by the imaging unit 204, corresponds to the range of the user's field of view.

[0046] In step S402, the system control unit 208 controls the user state estimation unit 165 so as to estimate the user state based on the eyeball information acquired in step S401.

[0047] In step S403, the system control unit 208 reads the predetermined conditions for detecting the recording timing (enabling the highlight recording flag) set in the timing detection unit 167. In the first embodiment, the following four conditions are assumed.

[0048] The first is the excitement (preference) condition (first condition). The excitement condition is a condition aimed at detecting the moment when the user sees something that matches their preference (something that excites them). The second is the attention condition (second condition). The attention condition is a condition aimed at detecting the moment when the user is paying attention. The third is the tracking condition (third condition). The tracking condition is a condition aimed at detecting the moment when the user is paying attention to a specific object and following it with their eyes. The fourth is the bird's-eye view condition (fourth condition). The bird's-eye view condition is a condition aimed at detecting the moment when the user is looking down (looking at the entire field of view).

[0049] In step S404, the system control unit 208 determines whether or not an excitation condition is set. If an excitation condition is set, the system control unit 208 proceeds to step S405, and if an excitation condition is not set, the system control unit 208 proceeds to step S406.

[0050] In step S405, system control unit 208 determines whether the user's state satisfies an excitement condition. The excitement condition is satisfied when the excitement level estimated by user state estimation unit 165 in step S402 is equal to or greater than a first threshold. If the excitement level is equal to or greater than the first threshold, system control unit 208 proceeds to step S413, and if the excitement level is less than the first threshold, system control unit 208 proceeds to step S406.

[0051] The user state estimation unit 165 estimates the excitement level, for example, taking into account the frequency of occurrence of microsaccades. The user state estimation unit 165 estimates a higher excitement level value the more frequently microsaccades occur in a given time period. Note that the user state estimation unit 165 may estimate the excitement level, for example, taking into account the amount of change in pupil diameter. The user state estimation unit 165 may estimate a higher excitement level value the more the amount of change in pupil diameter in a given time period.

[0052] In step S406, the system control unit 208 determines whether or not a condition of interest has been set. If a condition of interest has been set, the system control unit 208 proceeds to step S40. If the condition of interest is not satisfied, the process proceeds to step S408.

[0053] In step S407, system control unit 208 determines whether the user's state satisfies an attention condition. The attention condition is satisfied when the size of the gaze range estimated by user state estimation unit 165 in step S402 has decreased by an amount of change greater than the second threshold over a predetermined time period. If the size of the gaze range has decreased by an amount of change greater than the second threshold, system control unit 208 proceeds to step S413, and if the size has not decreased by an amount of change greater than the second threshold, system control unit 208 proceeds to step S408. Note that system control unit 208 may perform control such that if the size of the gaze range has decreased below a predetermined size, the process proceeds to step S413, and if the size has not decreased below the predetermined size, the process proceeds to step S408.

[0054] Here, the wider the gaze range, the larger the amplitude of microsaccades and the higher the oscillatory rate (lower the attenuation rate). Also, the wider the gaze range, the higher the frequency of microsaccades occurring in a given time. On the other hand, the narrower the gaze range, the smaller the amplitude of microsaccades and the lower the oscillatory rate (higher the attenuation rate). Also, the narrower the gaze range, the lower the frequency of microsaccades occurring in a given time. Therefore, the size of the gaze range can be estimated based on the amplitude, oscillatory rate, and frequency of microsaccades occurring.

[0055] In step S408, the system control unit 208 determines whether or not a tracking condition has been set. If a tracking condition has been set, the system control unit 208 proceeds to step S409, and if a tracking condition has not been set, the system control unit 208 proceeds to step S410.

[0056] In step S409, the system control unit 208 determines whether the user's state satisfies a tracking condition. The tracking condition is satisfied when the size of the gaze range estimated by the user state estimation unit 165 in step S402 is equal to or less than a third threshold, and the saccade speed acquired by the event data calculation unit 164 in step S401 is equal to or greater than a fourth threshold. If no saccade is observed, the system control unit 208 may determine that the speed is not equal to or greater than the fourth threshold. If the size of the gaze range is equal to or less than the third threshold and the saccade speed is equal to or greater than the fourth threshold, the system control unit 208 proceeds to step S413. If the size of the gaze range is equal to or less than the third threshold and the saccade speed is not equal to or greater than the fourth threshold, the system control unit 208 proceeds to step S410. The third and fourth thresholds may be absolute values ​​or relative ratios.

[0057] In step S410, the system control unit 208 determines whether or not an overhead view condition is set. If an overhead view condition is set, the system control unit 208 proceeds to step S411, and if an overhead view condition is not set, the system control unit 208 proceeds to step S412.

[0058] In step S411, the system control unit 208 determines whether the user's state satisfies the bird's-eye view condition. The bird's-eye view condition is satisfied when the size of the gaze range estimated by the user state estimation unit 165 in step S402 is greater than a fifth threshold for a predetermined time. If the size of the gaze range is greater than the fifth threshold for a predetermined time, the system control unit 208 proceeds to step S413; if the size is smaller than the fifth threshold, the system control unit 208 proceeds to step S412. The fifth threshold may be an absolute value or a relative ratio.

[0059] In step S412, the system control unit 208 outputs to the timing detection unit 167 an instruction to disable the highlight recording flag (a flag indicating the timing of recording a highlight).

[0060] In step S413, the system control unit 208 enables the highlight recording flag. The system control unit 208 outputs an instruction to the timing detection unit 167. The system control unit 208 repeats the processes of steps S401 to S413 while the power of the camera 100 is on. Note that, in the example of Fig. 4, the system control unit 208 enables the highlight recording flag when at least one condition is satisfied, but the system control unit 208 may also enable the highlight recording flag when multiple conditions are satisfied simultaneously.

[0061] 4, the system control unit 208 uses the excitement condition, attention condition, tracking condition, and bird's-eye view condition, but other conditions may also be used. For example, the system control unit 208 may use a condition related to the user's concentration level or fatigue level, and activate a highlight recording flag when the condition is satisfied.

[0062] The concentration level indicates the degree to which the user is concentrating on what they are looking at. The fatigue level is a value like the inverse of the concentration level and indicates the degree to which the user is not concentrating (fatigued). Here, the higher the user's fatigue level, the lower the frequency of microsaccades and the higher the frequency of blinking. Therefore, the fatigue level can be estimated based on the frequency of microsaccades and the frequency of blinking, etc. The system control unit 208 may, for example, control the highlight recording flag to be enabled when the fatigue level is equal to or greater than a sixth threshold. Furthermore, for example, the system control unit 208 may control the highlight recording flag to be enabled when it is estimated that the user's state has switched from a fatigued state to a concentrated state.

[0063] For example, in step S405, the system control unit 208 calculates the excitement level from the frequency of occurrence of microsaccades and switches processing depending on whether the excitement level is greater than the first threshold. However, this is not limited to this. For example, the system control unit 208 may determine whether the user is viewing something that suits their preferences based on the frequency of occurrence of microsaccades without calculating the excitement level. Furthermore, the system control unit 208 may calculate parameters other than the excitement level (such as the gaze level, concentration level, fatigue level, and bird's-eye view level) and switch processing depending on whether the calculated parameters are greater than a threshold. The system control unit 208 may determine whether the user is paying attention or concentrating based on the frequency of occurrence of microsaccades without calculating the gaze level, concentration level, fatigue level, and bird's-eye view level. As described above, the gaze level is an index that is higher the narrower the gaze range and lower the wider the range, and the bird's-eye view level is an antonym of the gaze level.

[0064] By setting the highlight recording flag as described above, it is possible to detect the timing of highlight recording based on the user's state.

[0065] <Highlight still image recording process> Next, the recording process for recording a highlight still image will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the recording process for a highlight still image. Each process in the flowchart of Fig. 5 is realized by the system control unit 208 loading a program stored in the nonvolatile memory 211 into the system memory 212, executing it, and controlling each functional block. For example, the recording process starts when the power of the camera 100 is turned on.

[0066] In step S501, the system control unit 208 drives the imaging unit 204 (image sensor) to output an analog signal.

[0067] In step S502, the system control unit 208 converts the analog signal output from the imaging unit 204 in step S501 into a digital signal using the A / D converter 205. The system control unit 208 records the digital signal in the memory 209 as still image data (image data) via the image processing unit 206 or the memory control unit 207.

[0068] In step S503, the system control unit 208 determines whether the highlight recording flag is enabled. If the highlight recording flag is enabled, the system control unit 208 proceeds to step S504, and if the highlight recording flag is disabled, the system control unit 208 proceeds to step S505.

[0069] In step S 504 , the system control unit 208 records the still image data recorded in the memory 209 onto the recording medium 103 .

[0070] In step S505, the system control unit 208 deletes the cache of the still image data recorded in the memory 209. After deleting the cache of the still image data, the system control unit 208 returns to step S501 and repeats the process.

[0071] In this way, the camera 100 can record the still image at the timing when the highlight recording flag is detected as a highlight still image.

[0072] <Highlight video recording process> Next, a recording process for recording a highlight video will be described with reference to Figs. 6 and 7. Fig. 6 is a flowchart showing the recording process for a highlight video. Fig. 7 is a diagram showing an example of a highlight video. Each process in the flowchart of Fig. 6 is realized by the system control unit 208 loading a program stored in the nonvolatile memory 211 into the system memory 212, executing it, and controlling each functional block. For example, the recording process starts when the power of the camera 100 is turned on. Note that the process of step S601 is the same as the process of step S503. The processes of steps S602 and S606 are the same as the process of step S501. The processes of steps S605 and S607 are the same as the process of step S502.

[0073] In step S603, the system control unit 208 compares the number of frames (images) cached in the memory 209 with the number of pre-detection frames set in the system memory 212. If the number of cached frames matches the number of pre-detection frames, the system control unit 208 proceeds to step S604, and if they do not match, the system control unit 208 proceeds to step S605.

[0074] Here, in order to record a video of a predetermined period including the timing when the highlight recording flag is enabled as a highlight video, the system control unit 208 controls the memory 209 to cache a certain number of frames. In FIG. 6, the video of the predetermined period includes frames before and after the timing when the highlight recording flag is enabled. Hereinafter, frames included in the highlight video that occur before the timing when the highlight recording flag is enabled will be referred to as pre-detection frames. The above-mentioned number of pre-detection frames is the number of pre-detection frames. Frames included in the highlight video that occur after the timing when the highlight recording flag is enabled will be referred to as post-detection frames, and the number of post-detection frames will be referred to as the number of post-detection frames. The number of pre-detection frames and the number of post-detection frames are changeable and are set in the system memory 212 by the system control unit 208. Note that the number of pre-detection frames and the number of post-detection frames may be values ​​that can be specified by the user. Note that the number of pre-detection frames and the number of post-detection frames may be the same value or different values.

[0075] In step S604, the system control unit 208 deletes the oldest frame from among the frames cached in the memory 209. In this way, the system control unit 208 maintains a state in which a certain number of frames, including the most recent frame, are cached in the memory 209.

[0076] In step S608, system control unit 208 compares the number of frames cached in memory 209 with the total number of frames in the highlight video (the sum of the number of pre-detection frames and the number of post-detection frames set in system memory 212). System control unit 208 repeats the processes of steps S606 and S607 until the number of cached frames matches the total number of frames in the highlight video, and if they match, proceeds to step S609.

[0077] In step S609, the system control unit 208 generates video data from the data (image data) of the multiple frames cached in the memory 209, and records the video data on the recording medium 103.

[0078] In step S610, the system control unit 208 deletes the image data cached in the memory 209. After completing the process of step S610 or step S605, the system control unit 208 returns to step S601 and repeats the recording process.

[0079] 6, once the system control unit 208 has recorded a highlight moving image with a predetermined number of frames on the recording medium 103, it deletes the image data cached in the memory 209 (steps S608 to S610). Note that the system control unit 208 may cache image data until the free space in the memory 209 reaches a predetermined capacity, and delete the image data cached in the memory 209 when the free space in the memory 209 falls below the predetermined capacity. This allows control so that, even when the highlight recording flag is repeatedly enabled and disabled, not only frames after the highlight recording flag is enabled but also frames before the flag is enabled are included in the highlight moving image.

[0080] Furthermore, when the number of cached frames matches the total number of frames in the highlight video, system control unit 208 generates a highlight video and records it on recording medium 103 (steps S608 to S609). Note that system control unit 208 may perform control such that, after a plurality of image groups have been accumulated in memory 209, a plurality of highlight videos corresponding to the plurality of image groups are generated all at once and recorded on recording medium 103.

[0081] An example of a highlight video will be described with reference to Fig. 7. In Fig. 7, as an example, the highlight recording flag is enabled when the tracking condition (the size of the gaze range is equal to or smaller than the third threshold and the saccade speed is equal to or larger than the fourth threshold) is satisfied.

[0082] Frames 701 to 708 show a subject 700 (a dog in FIG. 7). Areas 711 to 718 indicate the user's gaze range in frames 701 to 708. The size of areas 711 to 718 is equal to or smaller than a third threshold. Frames 701 to 708 are associated with timestamps 721 to 728.

[0083] In frames 701 to 704, ranges 711 to 714 are detected in approximately the same region. In frames 701 to 704, the system control unit 208 determines that the user is gazing at the subject 700 but that no saccade has occurred. Therefore, in frames 701 to 704 (timestamps 721 to 724), the system control unit 208 determines that the user's state does not satisfy the tracking conditions and disables the highlight recording flag (step S412).

[0084] In frame 705, the subject 700 moves, and the area in which the gaze range is detected moves from range 714 in frame 704 to range 715 in frame 705. In frame 705, the system control unit 208 determines that the speed of the saccade that occurred is equal to or greater than the fourth threshold. Therefore, in frame 705 (timestamp 725), the system control unit 208 It is determined that the state of the user satisfies the tracking condition, and the highlight recording flag is enabled (S step 413).

[0085] In FIG. 7, as an example, the number of pre-detection frames and the number of post-detection frames are set to 3. Frames 702 to 704 are frames cached in memory 209 as pre-detection frames 730 (the three frames immediately before frame 705 for which the highlight recording flag is enabled). Frames 705 to 707 are frames cached in memory 209 as post-detection frames 731 (the three frames after frame 705 for which the highlight recording flag is enabled). System control unit 208 records pre-detection frame 730 and post-detection frame 731 together as a highlight moving image on recording medium 103 (step S609). In this way, camera 100 can record a highlight moving image for a predetermined period that includes the timing when the highlight recording flag is enabled.

[0086] The system control unit 208 may record the frame for which the highlight recording flag is enabled as a still image on the recording medium 103, associating the pre-detection frame 730 with the post-detection frame 731.

[0087] The system control unit 208 may perform the highlight recording flag setting process shown in FIG. 4 in response to detecting that the user has placed their eye close to the subject, or may perform the highlight recording flag setting process in response to starting video recording. Furthermore, in the case of video recording, information corresponding to the highlight recording flag may be recorded in association with a frame. This allows the user to select a frame for which the highlight recording flag is enabled from among multiple frames in the video. For example, after recording a video, the user can select a frame for which the highlight recording flag is enabled to generate a highlight still image or a highlight video. The user may also be able to select (specify) and play back a frame for which the highlight recording flag is enabled during video playback. This allows the user to specify and play back an exciting scene, such as a soccer shot, or to fast-forward and play back an exciting scene.

[0088] In the first embodiment, the camera 100 estimates the state of the user based on the eyeball information of the user. The camera 100 records highlights when the estimated state of the user satisfies a predetermined condition. This allows the camera 100 to record highlights based on the state of the user.

[0089] <Example 2> In Example 2, an example in which the present invention is applied to a head-mounted display (HMD) will be described. When a user is concentrating on content displayed on the head-mounted display, it is difficult for the user to specify scenes for which highlights are to be recorded. Therefore, the present invention is applied to a head-mounted display. In this way, even if the user is concentrating on content displayed on the head-mounted display, highlights can be automatically recorded according to the user's state. Below, items common to Example 1 will be omitted, and differences will be mainly described.

[0090] <Configuration explanation> 8 is a block diagram showing an example of the configuration of a head-mounted display. A display unit 801 is a display unit that displays information to a user wearing the head-mounted display 800.

[0091] The display information displayed on the display unit 801 is transmitted from the recording medium 103 via the recording medium interface 226 or from the network 225 via the network interface 222. 4 into memory 209. System control unit 208 controls content such as still images and videos included in the display information loaded into memory 209 so that they are displayed on display unit 801. When a user views an image (still image or video) displayed on display unit 801, the range of the image displayed on display unit 801, i.e., the range of the image controlled by system control unit 208 to be displayed on display unit 801, corresponds to the range of the user's field of view.

[0092] The operation unit 802 is an input unit that accepts operations from the user (user operations), and is used to input various operational instructions to the system control unit 208.

[0093] The method for detecting the recording timing based on the eyeball information and state of the user in the head mounted display 800 is similar to the detection method used by the camera 100 described with reference to FIG. 4, and therefore will not be described here.

[0094] <Highlight still image recording process> Next, a recording process for recording a highlight still image will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the recording process for a highlight still image. Each process in the flowchart of Fig. 9 is realized by the system control unit 208 expanding a program stored in the nonvolatile memory 211 into the system memory 212, executing it, and controlling each functional block. For example, the recording process starts when the head mounted display 800 is turned on. The recording process may also start when the user puts on the head mounted display 800.

[0095] In step S 901 , the system control unit 208 reads display information from the recording medium 103 via the recording medium interface 226 or from the network 225 via the network interface 224 .

[0096] In step S902, the system control unit 208 records the display information read in step S901 in the memory 209 as still image data (image data).

[0097] In step S903, the system control unit 208 displays on the display unit 801 the still image data recorded in the memory 209 in step S902.

[0098] In step S904, the system control unit 208 determines whether the highlight recording flag is enabled. If the highlight recording flag is enabled, the system control unit 208 proceeds to step S905, and if the highlight recording flag is disabled, the system control unit 208 proceeds to step S906.

[0099] In step S 905 , the system control unit 208 records the still image data recorded in the memory 209 onto the recording medium 103 .

[0100] In step S906, the system control unit 208 deletes the cache of the still image data recorded in the memory 209. After deleting the cache of the still image data, the system control unit 208 returns to step S901 and repeats the process.

[0101] In this way, the head mounted display 800 can record the still image at the timing when the highlight recording flag is detected as a highlight still image.

[0102] <Highlight video recording process> Next, a recording process for recording a highlight video will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the recording process for a highlight video. Flowchart of Fig. 10 Each process in is realized by the system control unit 208 expanding a program stored in the nonvolatile memory 211 into the system memory 212, executing it, and controlling each functional block. For example, the recording process starts when the head mounted display 800 is turned on. The recording process may also start when the user puts on the head mounted display 800. Note that the process of step S1001 is the same as the process of step S904. The processes of steps S1004 to S1006 and steps S1007 to S1009 are the same as the processes of steps S901 to S903.

[0103] In step S1002, the system control unit 208 compares the number of frames cached in the memory 209 with the number of pre-detection frames set in the system memory 212. If the number of cached frames matches the number of pre-detection frames, the system control unit 208 proceeds to step S1003, and if they do not match, the system control unit 208 proceeds to step S1004.

[0104] In step S1010, system control unit 208 compares the number of frames cached in memory 209 with the total number of frames in the highlight video (the sum of the number of pre-detection frames and the number of post-detection frames set in system memory 212). System control unit 208 repeats the processes of steps S1007 to S1009 until the number of cached frames matches the total number of frames in the highlight video, and if they match, proceeds to step S1011.

[0105] In step S1011, the system control unit 208 generates video data from data (image data) of a plurality of frames cached in the memory 209, and records the video data on the recording medium 103.

[0106] In step S1012, the system control unit 208 deletes the image data cached in the memory 209. After completing the process of step S1006 or S1012, the system control unit 208 returns to step S1001 and repeats the recording process.

[0107] In this way, the head mounted display 800 can record a highlight moving image for a predetermined period including the timing at which the highlight recording flag is detected.

[0108] In the second embodiment, the head mounted display 800 estimates the state of the user based on the eyeball information of the user. The head mounted display 800 records highlights when the estimated state of the user satisfies a predetermined condition. This makes it possible to record highlights based on the state of the user.

[0109] <Other Examples> While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various modifications within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0110] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0111] 100: Camera 167: Timing detector 208: System control unit 209: Memory

Claims

1. An acquisition means for acquiring eyeball information of a user; an estimation means for estimating a size of the user's gaze range based on the eyeball information acquired by the acquisition means, and estimating whether the user is following a specific object with their eyes based on the eyeball information and the size of the gaze range; a recording means for recording an image corresponding to the user's field of view when the estimation means estimates that the user is tracking a specific object with his or her eyes; and the estimation means estimates that the user is tracking a specific object with his or her eyes when the size of the gaze range is equal to or smaller than a third threshold and the speed of the saccade is equal to or larger than a fourth threshold. An electronic device characterized by:

2. the recording means records, as images corresponding to the user's field of view, a plurality of images taken during a predetermined period including a timing at which the user is estimated to be tracking a specific object with his or her eyes; The electronic device according to claim 1 .

3. the predetermined period includes a timing after a timing at which the user is estimated to be tracking a specific object with his / her eyes. The electronic device according to claim 2 .

4. the predetermined period includes a timing before the timing at which the user is estimated to be tracking a specific object with his or her eyes.

4. The electronic device according to claim 2 or 3.

5. An acquisition means for acquiring user eyeball information; an estimation means for estimating a size of the user's gaze range based on the eyeball information acquired by the acquisition means, and estimating whether the user is looking down based on the size of the gaze range; a recording means for recording an image corresponding to the user's field of view when the estimation means estimates that the user is looking down; and The estimation means estimates that the user is looking down when the size of the gaze range is larger than a fifth threshold for a predetermined time. An electronic device characterized by:

6. the estimation means further estimates whether the user is tracking a specific object with their eyes based on the eyeball information and the size of the gaze range; the recording means records an image corresponding to the user's field of view when the estimating means estimates that the user is tracking a specific object with his or her eyes. The electronic device according to claim 5 .

7. the estimation means estimates that the user is tracking a specific object with his or her eyes when the size of the gaze range is equal to or smaller than a third threshold and the speed of the saccade is equal to or larger than a fourth threshold.

7. The electronic device according to claim 6.

8. The eyeball information includes at least one of a frequency of occurrence of microsaccades, a size of microsaccades, a gaze position, a direction of saccades, a speed of saccades, a size of pupils, a change in pupil diameter, a speed of blinks, and a frequency of blinks. The electronic device according to claim 1 .

9. The estimation means further estimates whether the user is viewing something that matches his or her preferences based on the eyeball information, the recording means records an image corresponding to the user's field of view when the estimation means estimates that the user is viewing something that matches his or her preferences. The electronic device according to claim 1 .

10. the estimation means further estimates an excitement level of the user based on the eyeball information, and estimates whether the user is watching something that suits his or her preferences based on the excitement level; The estimation means estimates that the user is watching something that suits his or her preferences when the excitement level is equal to or greater than a first threshold.

10. The electronic device according to claim 9.

11. The estimation means further estimates whether the user is paying attention based on the size of the gaze range, the recording means records an image corresponding to the user's field of view when the estimation means estimates that the user is in a state of paying attention. The electronic device according to claim 1 .

12. The estimation means estimates that the user is paying attention when the size of the gaze range decreases by an amount of change greater than a second threshold value over a predetermined time period. The electronic device according to claim 11.

13. An imaging means; and a control means for controlling the display means to display the image captured by the imaging means. The acquisition means views an image captured by the imaging means and displayed on the display means. Obtain the user's eyeball information, the recording means records the image captured by the imaging means as an image corresponding to the user's field of view. The electronic device according to claim 1 .

14. further comprising an imaging means for imaging a range corresponding to the user's field of view, the recording means records the image captured by the imaging means as an image corresponding to the user's field of view. The electronic device according to claim 1 .

15. The apparatus further includes a control means for controlling the display means to display the moving image, the acquiring means acquires eyeball information of a user watching the moving image displayed on the display means, the recording means records an image of a frame of the moving image as an image corresponding to the user's field of view; The electronic device according to claim 1 .

16. An acquisition step of acquiring eyeball information of a user; an estimation step of estimating a size of the user's gaze range based on the eyeball information acquired in the acquisition step, and estimating whether the user is tracking a specific object with their eyes based on the eyeball information and the size of the gaze range; a recording step of recording an image corresponding to the user's field of view when it is estimated in the estimating step that the user is tracking a specific object with their eyes; and In the estimation step, when the size of the gaze range is equal to or smaller than a third threshold and the speed of the saccade is equal to or larger than a fourth threshold, it is estimated that the user is tracking a specific object with their eyes. A method for controlling an electronic device.

17. An acquisition step of acquiring eyeball information of a user; an estimation step of estimating a size of the user's gaze range based on the eyeball information acquired in the acquisition step, and estimating whether the user is looking down based on the size of the gaze range; a recording step of recording an image corresponding to the user's field of view when it is estimated in the estimating step that the user is looking down; and In the estimating step, it is estimated that the user is looking down when the size of the gaze range is larger than a fifth threshold for a predetermined time. A method for controlling an electronic device.

18. 18. A program for causing a computer to execute each step of the method for controlling an electronic device according to claim 16 or 17.

19. 18. A computer-readable recording medium storing a program for causing a computer to execute each step of the method for controlling an electronic device according to claim 16 or 17.

Images