Imaging apparatus

Abstract

To provide a technique to allow focusing on a user's favorite subject with high accuracy (constantly).SOLUTION: An imaging apparatus of the present invention has: first acquisition means that acquires line-of-sight information related to the line of sight of a user; second acquisition means that acquires eyeball information related to the state of the eyeball of the user; detection means that detects a subject; determination means that determines a subject to which the line of sight of the user is directed on the basis of the line-of-sight information and a result of detection of the subject, and determines an evaluation value of the determined subject on the basis of the eyeball information; and processing means that performs autofocus processing on the basis of the evaluation value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an imaging device, and more particularly to control based on a user's line of sight. [Background technology]

[0002] Users can focus on a subject by aligning the displayed autofocus (AF) frame with the subject. Some imaging devices can also detect a subject using a face detection function and track the subject while maintaining focus on that subject. However, this method can make it difficult to focus on the user's preferred subject when multiple subjects are detected, such as when photographing a soccer game.

[0003] Patent Document 1 discloses a technique for focusing on a subject that a user is looking at. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-034569 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology disclosed in Patent Document 1, if the user directs their gaze to a subject that is not their preference, the subject will be focused on. For example, if the user directs their gaze to a desired subject and focuses on the desired subject, if an obstacle appears between the user and the desired subject (if the line of sight is blocked by the obstacle), the focus target will switch from the desired subject to the obstacle.

[0006] The present invention aims to provide a technology that enables a user to (always) focus on a subject of their choice with high accuracy. [Means for solving the problem]

[0007] The imaging device of the present invention is characterized by having a first acquisition means for acquiring gaze information regarding a user's gaze, a second acquisition means for acquiring eyeball information regarding the state of the user's eyeballs, a detection means for detecting a subject, a determination means for determining a subject at which the user's gaze is directed based on the gaze information and the subject detection result, and determining an evaluation value of the determined subject based on the eyeball information, and a processing means for performing autofocus processing based on the evaluation value. [Effects of the Invention]

[0008] According to the present invention, it is possible to (always) focus on a subject that the user prefers with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an external view of a digital camera. [Figure 2] FIG. 1 is a block diagram of a digital camera. [Figure 3] 10 is a flowchart showing a favorite shooting mode process according to the first embodiment. [Figure 4] FIG. 2 is a schematic diagram showing an example of pupil diameter. [Figure 5] 10 is a graph showing an example of a change in pupil diameter over time. [Figure 6] FIG. 2 is a schematic diagram showing an example of a captured image. [Figure 7] 10 is a flowchart showing a favorite shooting mode process according to the second embodiment. [Figure 8] 11 is a flowchart showing a favorite shooting mode process according to the third embodiment. [Figure 9] FIG. 10 is a schematic diagram showing an example of display of items. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Figures 1(A) and 1(B) are external views of a digital camera 100 (image capture device) as an example of a device (electronic device) to which the present invention can be applied. Figure 1(A) is a front perspective view of the digital camera 100, and Figure 1(B) is a rear perspective view of the digital camera 100.

[0011] The display unit 28 is a display unit provided on the back surface of the digital camera 100, and displays images and various information. The touch panel 70a can detect touch operations on the display surface (touch operation surface) of the display unit 28. The out-of-viewfinder display unit 43 is a display unit provided on the top surface of the digital camera 100, and displays various settings of the digital camera 100, including the shutter speed and aperture. The shutter button 61 is an operation member for issuing shooting instructions. The mode selector switch 60 is an operation member for switching between various modes. The terminal cover 40 is a cover that protects a connector (not shown) for connecting a connection cable or the like that connects the digital camera 100 to an external device.

[0012] The main electronic dial 71 is a rotary operation member, and by turning the main electronic dial 71, settings such as shutter speed and aperture can be changed. The power switch 72 is an operation member that switches the power of the digital camera 100 on and off. The sub electronic dial 73 is a rotary operation member, and by turning the sub electronic dial 73, the selection frame (cursor) can be moved, images can be forwarded, etc. The four-way key 74 is configured so that the up, down, left, and right parts can each be pressed, and processing can be performed according to the part of the four-way key 74 that is pressed. The SET button 75 is a push button, and is mainly used to confirm a selection item, etc.

[0013] The movie button 76 is used to start or stop movie shooting (recording). The AE lock button 77 is a push button, and pressing the AE lock button 77 in shooting standby mode fixes the exposure state. The enlarge button 78 is an operation button for switching the enlargement mode on and off in the live view display (LV display) in shooting mode. By turning the enlargement mode on and operating the main electronic dial 71, the live view image (LV image) can be enlarged or reduced. In playback mode, the enlargement button 78 functions as an operation button for enlarging the playback image or increasing its magnification. The playback button 79 is an operation button for switching between shooting mode and playback mode. Pressing the playback button 79 in shooting mode switches to playback mode, and the most recent image recorded on the recording medium 200 (described below) can be displayed on the display unit 28 or EVF 29 (described below; electronic viewfinder). Menu button 80 is a push button used to perform an instruction operation to display a menu screen, and when menu button 80 is pressed, a menu screen on which various settings can be made is displayed on display unit 28 or EVF 29. The user can intuitively make various settings using the menu screen displayed on display unit 28 or EVF 29, four-way key 74, and SET button 75.

[0014] The communication terminal 10 is a communication terminal through which the digital camera 100 communicates with the lens unit 150 (described later; detachable). The eyepiece 16 is the eyepiece of an eyepiece finder (a peer-type finder), and the user can view the image displayed on the internal EVF 29 through the eyepiece 16. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether or not the user (photographer) has placed their eye on the eyepiece 16. The lid 202 is a lid for a slot that stores a recording medium 200 (described later). The grip unit 90 is a lid that allows the user to hold the digital camera The grip section 90 is shaped to be easy to hold with the right hand when holding the digital camera 100. When the digital camera 100 is held by gripping the grip section 90 with the little finger, ring finger, and middle finger of the right hand, the shutter button 61 and main electronic dial 71 are positioned so that they can be operated with the index finger of the right hand. In the same position, the sub electronic dial 73 is positioned so that it can be operated with the thumb of the right hand.

[0015] FIG. 2 is a block diagram showing an example configuration of a digital camera 100. The lens unit 150 is a lens unit equipped with an interchangeable photographic lens. The lens 103 is usually composed of multiple lenses, but FIG. 2 shows only one lens for simplicity's sake. The communication terminal 6 is a communication terminal through which the lens unit 150 communicates with the digital camera 100, and the communication terminal 10 is a communication terminal through which the digital camera 100 communicates with the lens unit 150. The lens unit 150 communicates with the system controller 50 via these communication terminals 6 and 10. The lens unit 150 controls the aperture 1 via the aperture drive circuit 2 by the internal lens system control circuit 4. The lens unit 150 also adjusts the focus by displacing the position of the lens 103 via the AF drive circuit 3 by the lens system control circuit 4.

[0016] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

[0017] The imaging unit 22 is an imaging element (image sensor) formed of a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The A / D converter 23 converts the analog signal output from the imaging unit 22 into a digital signal. The imaging unit 22 captures an image in synchronization with a horizontal synchronization signal and a vertical line synchronization signal output from a timing generator (not shown), and the imaging unit 22 outputs one frame of image data as frame data at the cycle of the vertical line synchronization signal. While an event sensor 163, which will be described later, is an event-based vision sensor (an asynchronous event-based sensor), the imaging unit 22 is a synchronous frame-based sensor.

[0018] The image processing unit 24 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on data from the A / D converter 23 or data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows for TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, EF (pre-flash) processing, etc. AF modes include a one-shot AF mode that fixes the focus on a predetermined area and a servo AF mode that continuously focuses on a predetermined subject, which the user can switch between at will by operating the operation unit 70. These AF modes can also be switched automatically (AI focus AF).

[0019] The image processing unit 24 further performs predetermined calculations using the captured image data and performs TTL-type AWB (auto white balance) processing based on the calculation results. In the case of still image capture, the image data is encoded in a recording format such as JPEG to generate a still image file. In the case of video capture, video data encoded in H.264 / AVC or the like is multiplexed with audio data input from a microphone (not shown), and the multiplexed data is shaped into a video format set as the recording format to generate a video file.

[0020] The memory control unit 15 controls the transmission and reception of data between the A / D converter 23, the image processing unit 24, and the memory 32. The output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15. Alternatively, the output data from the A / D converter 23 is written to the memory 32 via the memory control unit 15 without going through the image processing unit 24. The memory 32 stores the digital data obtained by the imaging unit 22 and converted by the A / D converter 23 into digital data. The memory 32 stores image data captured by the camera, as well as image data to be displayed on the display unit 28 and EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined length of video and audio.

[0021] The memory 32 also serves as a memory (video memory) for displaying images. The display image data written to the memory 32 is displayed on the display unit 28 or EVF 29 via the memory control unit 15. The display unit 28 and EVF 29 are each a display such as an LCD or organic EL display, and perform display in response to a signal from the memory control unit 15. Live view display (LV display) can be performed by sequentially transferring and displaying data that has been A / D converted by the A / D converter 23 and stored in the memory 32 to the display unit 28 or EVF 29. Hereinafter, an image displayed in live view display will be referred to as a live view image (LV image). The display unit 28 and EVF 29 have multiple display modes corresponding to the operating modes of the digital camera 100. For example, there is a shooting mode that performs LV display, a playback mode that plays back captured images and videos, and a MENU mode that allows various settings. The user can freely switch between display modes by operating the operation unit 70.

[0022] Various settings of the digital camera 100 such as shutter speed and aperture are displayed on the outside viewfinder display 43 via an outside viewfinder display drive circuit 44 .

[0023] The nonvolatile memory 56 is an electrically erasable and recordable memory, such as a Flash-ROM. Constants, programs, etc. for the operation of the system control unit 50 are recorded in the nonvolatile memory 56. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0024] The system control unit 50 is a control unit consisting of at least one processor and / or at least one circuit, and controls the entire digital camera 100. The system control unit 50 executes programs stored in the nonvolatile memory 56 to realize the various processes of this embodiment, which will be described later. The system memory 52 is, for example, a RAM, and the system control unit 50 loads constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like into the system memory 52. ​​The system control unit 50 also controls the display by controlling the memory 32, the display unit 28, the EVF 29, and the like. The system control unit 50 can also detect the operating mode and state of the digital camera 100 as a work process. Examples of operating modes of the digital camera 100 include still image capture mode SW1, still image capture mode SW2, video capture mode, live view mode, MENU mode, and sleep mode. Examples of states of the digital camera 100 include tripod-mounted, handheld, walking, running, and panning.

[0025] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0026] The operation unit 70 is an input unit that accepts operations from the user (user operations) and is used to input various operational instructions to the system control unit 50. As shown in Fig. 2, the operation unit 70 includes a mode selector switch 60, a shutter button 61, a power switch 72, a touch panel 70a, other operation members 70b, etc. The other operation members 70b include a main electronic dial 71, a sub electronic dial 73, a four-way key 74, a SET button 75, a video button 76, an AE lock button 77, a magnification button 78, a playback button 79, a menu button 80, etc.

[0027] The mode changeover switch 60 switches the operation mode of the system control unit 50 to one of still image shooting mode, video shooting mode, playback mode, etc. The modes include auto shooting mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide shooting settings for different shooting scenes. The user can directly switch to one of these modes using the mode selector switch 60. Alternatively, after first switching to a list screen of shooting modes with the mode selector switch 60, the user may selectively switch to one of the displayed modes using another operating member. Similarly, the video shooting mode may also include multiple modes.

[0028] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 63. The first shutter switch 62 is turned ON when the shutter button 61 is pressed halfway (a shooting preparation command) and generates a first shutter switch signal SW1. The system control unit 50 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing in response to the first shutter switch signal SW1. The second shutter switch 63 is turned ON when the shutter button 61 is pressed fully (a shooting command) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of shooting processing operations in response to the second shutter switch signal SW2, from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file.

[0029] The touch panel 70a is a touch sensor that detects various touch operations on the display surface of the display unit 28 (the operation surface of the touch panel 70a). The touch panel 70a and the display unit 28 can be configured as an integrated unit. For example, the touch panel 70a is configured so that its light transmittance does not interfere with the display of the display unit 28, and is attached to the upper layer of the display surface of the display unit 28. Input coordinates on the touch panel 70a are associated with display coordinates on the display surface of the display unit 28. This makes it possible to provide a GUI (Graphical User Interface) that allows the user to directly operate the screen displayed on the display unit 28.

[0030] The power supply control unit 31 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the type of battery, and the remaining battery power. The power supply control unit 31 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or lithium-ion batteries, an AC adapter, etc.

[0031] The recording medium I / F 17 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0032] The communication unit 54 transmits and receives video signals and audio signals to and from external devices connected wirelessly or via a wired cable. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can receive image data and various other information from external devices.

[0033] The orientation detection unit 55 detects the orientation of the digital camera 100 relative to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is determined whether the image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. It is possible. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. It is also possible to detect the movement of the digital camera 100 (panning, tilting, lifting, whether or not it is stationary, etc.) using the acceleration sensor or gyro sensor that is the attitude detection unit 55.

[0034] The eyepiece detection unit 57 is an eyepiece detection sensor that detects (approach detection) whether the eye (object) 160 approaches (approach) or moves away (away) from (moves away from) the eyepiece 16 of the eyepiece finder (hereinafter simply referred to as the "finder"). The system control unit 50 switches the display unit 28 and the EVF 29 between on (display state) and off (non-display state) depending on the state detected by the eyepiece detection unit 57. More specifically, at least in a shooting standby state and when the display destination switching setting is automatic switching, when the eye is not in contact with the camera, the display is turned on with the display on the display unit 28 and the EVF 29 is hidden. When the eye is in contact with the camera, the display is turned on with the display on the EVF 29 and the display on the display unit 28 is hidden. For example, an infrared proximity sensor can be used as the eyepiece detection unit 57, and it can detect the approach of an object to the eyepiece 16 of the finder incorporating the EVF 29. When an object approaches, the infrared light emitted from the infrared light-emitting diode 58 is reflected by the object and received by the light-receiving section (not shown) of the infrared proximity sensor. The amount of received infrared light can also be used to determine the distance the object has come from the eyepiece 16 (eyepiece distance).

[0035] The subject identification unit 69 analyzes the image data obtained by the imaging unit 22, and identifies the type of subject and specifies the size and position within the image data. The subject identification unit 69 can use, for example, a convolutional neural network, which is widely used for image recognition.

[0036] The eyeball detection unit 161 is composed of an eyeball detection lens 162, an event sensor 163, and an event data calculation unit 164, which will be described later. The eyeball detection unit 161 is capable of acquiring eyeball information relating to the state of the eyeball (eye) 160 of the user looking through the viewfinder.

[0037] The infrared light emitted from the infrared light emitting diode 58 is reflected by the user's eyeball (eye) 160 , and the reflected infrared light passes through the eyeball detection lens 162 and forms an image on the imaging surface of the event sensor 163 .

[0038] The event sensor 163 is an event-based vision sensor that detects changes in the luminance of light incident on each pixel and outputs information about the pixel where the luminance change occurred asynchronously with other pixels. The data output from the event sensor 163 includes, for example, the position coordinates of the pixel where the luminance change (event) occurred, the polarity (positive or negative) of the luminance change, and timing information corresponding to the time the event occurred. This data will be referred to as event data hereafter. Compared to synchronous frame-based sensors such as the imaging unit 22, the event sensor 163 eliminates redundancy in the output information and is characterized by high-speed operation, a high dynamic range, and low power consumption. However, because the event data (information about pixels where the luminance change occurred) is output asynchronously with other pixels, special processing is required to determine the correlation between the event data. To determine the correlation between the event data, it is necessary to accumulate the event data output from the event sensor 163 over a predetermined period of time and perform various arithmetic operations on the results.

[0039] The event data calculation unit 164 is a calculation unit for acquiring (detecting) eyeball information based on event data that is continuously and asynchronously output from the event sensor 163. For example, the event data calculation unit 164 accumulates event data that occurs during a predetermined time period, and Eyeball information is acquired by processing multiple accumulated event data as a single set of data. By varying the accumulation time for accumulating event data, it is possible to acquire multiple pieces of eyeball information with different occurrence rates. Eyeball information includes, for example, gaze information regarding the user's line of sight, saccade information regarding the direction and speed of saccades, and microsaccade information regarding the frequency and amplitude of microsaccades (amount of change in gaze position). Gaze information is, for example, coordinate information indicating the position where the user is looking (gaze position) or angle information indicating the direction the user is looking (gaze direction). Eyeball information may also include information regarding eye movements other than saccades and microsaccades, pupil information regarding pupil size and changes therein, and blink information regarding the speed and frequency of blinks. These pieces of information are merely examples, and eyeball information is not limited to these pieces of information. The event data calculation unit 164 may map the event data for the accumulation time as one frame of image data based on event occurrence coordinates (the position coordinates of pixels where a luminance change (event) occurred), and perform image processing. With this configuration, eyeball information can be obtained from one frame of image data obtained by mapping event data for the accumulation time using frame-based image processing. Note that gaze information may be interpreted as information separate from eyeball information.

[0040] The user state determination unit 165 is a determination unit that determines the state of the user based on the eyeball information obtained by the event data calculation unit 164. For example, the size (width) of the gaze range or the degree of gaze (bird's-eye view degree) can be determined from the frequency and amplitude of microsaccades. Here, the gaze range is synonymous with the attention range or the focus range. The narrower the gaze range, the higher the gaze degree, and the wider it is, the lower the gaze degree. The bird's-eye view degree is defined as the antonym of the degree of gaze. Furthermore, the user's concentration level (state of concentration) or fatigue level can be determined from the frequency and amplitude of microsaccades, the size and change of the pupil, and the speed and frequency of blinks. Furthermore, the user's excitement level is related to the speed and diameter of the microsaccades and can be determined from both parameters. The excitement level is an index that increases when the user is looking at an object with a high preference (such as a favorite face) and decreases when the user is looking at an object with a low preference, so it can also be considered as a preference level. The user state determination unit 165 can be configured by a neural network that receives, for example, parameters related to eyeball information and the identification result of the subject identification unit 69 as input and outputs information related to the above-mentioned user state (hereinafter referred to as user state information). However, the configuration of the user state determination unit 165 is not limited to the above configuration. The eyeball information used by the user state determination unit 165 and the determination result of the user state determination unit 165 are not limited to those described above.

[0041] The gaze input setting unit 166 sets whether the processing of the eyeball detection unit 161 is enabled or disabled via the system control unit 50. The gaze input setting unit 166 can also set parameters and conditions related to the processing of the event data calculation unit 164 and the user state determination unit 165. For example, the user can arbitrarily set these settings from a menu screen or the like.

[0042] The system control unit 50 can also obtain information about the area of ​​the EVF 29 in which the imaged subject (object) is displayed and at what size. Furthermore, the eyeball detection unit 161 can also obtain information about the area of ​​the EVF 29 at which the user is directing their gaze. This allows the system control unit 50 to determine which area of ​​the subject the user is looking at.

[0043] Example 1 A first embodiment of the present invention will be described. FIG. 3 is a flowchart showing a favorite shooting mode process according to the first embodiment. The favorite shooting mode process in FIG. 3 is realized by the system control unit 50 expanding a program stored in the nonvolatile memory 56 into the system memory 52 and executing it. For example, when a favorite shooting mode is set, the system control unit 50 starts the favorite shooting mode process in FIG. 3. Note that FIG. 3 omits several operations, such as a shooting preparation operation and a shooting processing operation.

[0044] In step S301, the system control unit 50 controls the object identification unit 69 to detect an object from the image obtained by the imaging unit 22. This acquires object information indicating the type, size, position, etc. of the object in the image.

[0045] In step S302, the system control unit 50 controls the eyeball detection unit 161 to acquire eyeball information (for example, microsaccade information and pupil information) relating to the state of the user's eyeballs.

[0046] In step S303, the system control unit 50 controls the user state determination unit 165 based on the eyeball information acquired in step S302 to acquire the user's preference level.

[0047] 4(A) and 4(B) are schematic diagrams showing examples of pupil diameter. Although FIGS. 4(A) and 4(B) show the pupil diameters of the same person, pupil diameter 401 in FIG. 4(A) is smaller than pupil diameter 402 in FIG. 4(B). A person's pupil diameter depends on the person's state and the surrounding environment. FIG. 5 is a graph showing an example of changes in pupil diameter over time. When a person looks at an object, the person's pupil diameter first decreases and then increases, as shown by curves 501 and 502 in FIG. 5. Curve 501 shows the change in pupil diameter when the person looks at an object they do not like, and curve 502 shows the change in pupil diameter when the person looks at an object they like. As shown by curves 501 and 502, when the person looks at an object they do not like, the change (amplitude) in pupil diameter is larger than when the person looks at an object they do not like. The system control unit 50 can determine the pupil diameter and the amount of change in pupil diameter from the pupil information included in the eyeball information.

[0048] In step S303, for example, the system control unit 50 determines a preference level that is higher the greater the change in pupil diameter based on the pupil information included in the eyeball information. For example, a value between 0 and 100 is determined as the preference level. The preference level may be calculated by a mathematical formula or may be obtained using a lookup table or the like. When determining the preference level, changes in pupil diameter due to changes in ambient luminance may be taken into consideration.

[0049] In step S304, the system control unit 50 controls the eyeball detection unit 161 to acquire line-of-sight information relating to the user's line of sight (for example, information indicating the line-of-sight position or line-of-sight direction).

[0050] In step S305, the system control unit 50 determines the subject at which the user's gaze is directed, based on the subject information acquired in step S301 and the gaze information acquired in step S304. Then, the system control unit 50 associates the subject information of the determined subject with the preference level determined in step S303.

[0051] In step S306, the system control unit 50 updates the evaluation value of each subject stored in the memory 32 using the preference degree acquired in step S303. In the memory 32, subject information and evaluation values ​​are linked to each other. By determining whether or not subject information linked to the preference degree is stored in the memory 32 in step S305, it is possible to determine whether or not the evaluation value of the subject at which the user's gaze is directed is stored in the memory 32. Furthermore, if the evaluation value of the subject at which the user's gaze is directed is stored in the memory 32, it is also possible to grasp that evaluation value. If the evaluation value of the subject at which the user's gaze is directed is not stored in the memory 32, the system control unit 50 determines the preference degree acquired in step S303 as the evaluation value of the subject at which the user's gaze is directed. Then, the system control unit 50 stores the subject information of the subject at which the user's gaze is directed and the determined evaluation value (preference degree acquired in step S303) in the memory 32, linking them to each other. If the evaluation value of the subject at which the user's gaze is directed is stored in the memory 32 In this case, the system control unit 50 updates the evaluation value of the subject at which the user's gaze is directed by adding the preference level acquired in step S303 to the evaluation value.

[0052] As a result, the evaluation value of the subject increases as the time that the user directs their gaze at the subject increases. In the above example, the process of increasing the evaluation value of the subject by an amount of change based on eyeball information is repeated during the period that the user directs their gaze at the subject. A subject that the user looks at for a long time (a subject that the user continues to look at or repeatedly looks at) is likely to be a subject that the user likes. By increasing the evaluation value of the subject as the time that the user directs their gaze at the subject increases, the evaluation value of the subject that the user likes can be increased. Note that the method of increasing the evaluation value is not particularly limited. For example, after the preference level of the subject is determined as the evaluation value of the subject, 1 may be added to the evaluation value of the subject for every additional second that the user directs their gaze at the subject.

[0053] The system control unit 50 may reduce the evaluation value of a subject as the time the user does not direct their gaze toward the subject increases. For example, the system control unit 50 may repeat a process of reducing the evaluation value of the subject by a change amount based on the eyeball information during a period when the user does not direct their gaze toward the subject. In step S306, the system control unit 50 may update the evaluation value of the subject to which the user does not direct their gaze by subtracting the preference degree acquired in step S303 from the evaluation value. A subject that the user looks at for a short time is likely to be a subject that the user does not like. By reducing the evaluation value of the subject as the time the user does not direct their gaze toward the subject increases, the evaluation value of a subject that the user does not like can be reduced. Note that the method of reducing the evaluation value is not particularly limited. For example, after determining the preference degree of a subject as the evaluation value of the subject, the system control unit 50 may subtract 1 from the evaluation value of the subject for every additional second that the user does not direct their gaze toward the subject.

[0054] There is a high possibility that a subject included in the gaze range is a subject that the user likes. Therefore, the system control unit 50 estimates the user's gaze range based on the microsaccade information included in the eyeball information, and the evaluation value of the subject included in the gaze range does not need to be reduced.

[0055] The system control unit 50 may or may not reset the evaluation value of each subject for each scene. For example, in the current shooting scene, the system control unit 50 may use the evaluation value of the subject determined in a previous shooting scene as the initial value of the evaluation value of that subject.

[0056] The system control unit 50 may rearrange the evaluation values ​​of the subjects stored in the memory 32 in ascending or descending order.

[0057] Returning to the description of Fig. 3, in step S307, the system control unit 50 performs autofocus processing based on the evaluation value of each subject. For example, the system control unit 50 performs autofocus processing so as to focus on the subject with the highest evaluation value detected in step S301. However, this is not limiting, and the autofocus processing may be performed so as to focus on, for example, a predetermined number of subjects with the highest evaluation values.

[0058] In step S308, the system control unit 50 determines whether or not an instruction to end the favorite shooting mode processing has been issued. If an instruction to end the favorite shooting mode processing has been issued, the shooting mode processing is terminated, and if not, the process proceeds to S301. For example, if the mode changeover switch 60 has been operated, that is, if an instruction to switch to another operation mode has been issued, the system control unit 50 determines that an instruction to end the favorite shooting mode processing has been issued. Also, if the power switch 72 has been operated, that is, if an instruction to power off the digital camera 100 has been issued, the system control unit 50 0 determines that an instruction to end the favorite shooting mode processing has been issued.

[0059] 6(A) to 6(C) are schematic diagrams showing examples of captured images. The captured scene is a soccer game, and there are three players 601 to 603 and a ball 604. It is assumed that the user is interested in player 602, and the user is directing his or her gaze at player 602.

[0060] With conventional technology that focuses on a subject that the user's gaze is directed at, it is possible to focus on the athlete 602 at the timing shown in Figure 6(A). However, if an obstacle appears between the user and the athlete 602 (if the line of sight is blocked by the obstacle), the focus will be on the obstacle and the athlete 602 will no longer be in focus. For example, as shown in Figure 6(C), the focus will be on the referee 605 and the athlete 602 will no longer be in focus.

[0061] In the first embodiment, the focus target is determined using an evaluation value based on eyeball information, thereby solving the problems of the conventional technology. Because the user is interested in the athlete 602, a high evaluation value is assigned to the athlete 602, and it is unlikely that a higher evaluation value than the athlete 602 is assigned to other subjects. Therefore, at the timing shown in FIG. 6(A), the athlete 602 can be focused on, as with the conventional technology. Furthermore, even if an obstacle appears between the user and the athlete 602, the focus does not shift to the obstacle, and the focus can be maintained generally on the athlete 602, allowing the focus to be quickly shifted to the athlete 602 when the athlete 602 reappears. For example, as shown in FIG. 6(B), the focus does not shift to the judge 605, and the focus can be maintained generally on the athlete 602, allowing the focus to be quickly shifted to the athlete 602 when the athlete 602 reappears.

[0062] <Example 2> A second embodiment of the present invention will now be described. FIG. 7 is a flowchart showing a favorite shooting mode process according to the second embodiment. The favorite shooting mode process in FIG. 7 is realized by the system control unit 50 expanding a program stored in the nonvolatile memory 56 into the system memory 52 and executing it. For example, when a favorite shooting mode is set, the system control unit 50 starts the favorite shooting mode process in FIG. 7. Note that FIG. 7 omits several operations, such as a shooting preparation operation and a shooting processing operation.

[0063] Steps S701 to S704 are the same as steps S301 to S304 in the first embodiment (FIG. 3).

[0064] In step S705, the system control unit 50 determines whether the user is directing their gaze at a predetermined type of subject (e.g., a person) based on the subject information acquired in step S701 and the gaze information acquired in step S704. If the user is directing their gaze at a predetermined type of subject (e.g., a person), the system control unit 50 associates the subject information of the subject at which the user's gaze is directed with the preference level determined in step S703, and resets the measurement time of the system timer 53 to 0. Then, the system control unit 50 proceeds to step S706. If the user is not directing their gaze at a predetermined type of subject (e.g., a person), the system control unit 50 starts timing by the system timer 53 if it has not already started. Then, the system control unit 50 proceeds to step S708. In this way, the system timer 53 measures the duration during which the user is not directing their gaze at a predetermined type of subject (e.g., a person).

[0065] Steps S706 and S707 are the same as steps S306 and S307 in the first embodiment (FIG. 3).

[0066] In step S708, the system control unit 50 determines whether the time measured by the system timer 53 has reached a predetermined time T (whether the time T has elapsed while the user is not directing their gaze at a predetermined type of subject (e.g., a person)). If the time measured by the system timer 53 has reached the time T, the system control unit 50 resets the time measured by the system timer 53 to 0. Then, the process proceeds to step S709. If the time measured by the system timer 53 has not reached the time T, the process proceeds to step S710.

[0067] In step S709, the system control unit 50 reduces, by a predetermined reduction amount, the evaluation values ​​of all subjects stored in the memory 32. This allows the evaluation values ​​of past subjects to be reduced, and the next time a preferred subject appears, the preferred subject can be focused on smoothly (in a short time).

[0068] Step S710 is the same as step S308 in the first embodiment (FIG. 3).

[0069] Example 3 A third embodiment of the present invention will now be described. FIG. 8 is a flowchart showing a favorite shooting mode process according to the third embodiment. The favorite shooting mode process of FIG. 8 is realized by the system control unit 50 loading a program stored in the non-volatile memory 56 into the system memory 52 and executing it. For example, when the favorite shooting mode is set, the system control unit 50 starts the favorite shooting mode process of FIG. 8. Note that FIG. 8 omits several operations such as shooting preparation operations and shooting processing operations. In the favorite shooting mode, LV display is performed.

[0070] Steps S801 to S807 are the same as steps S701 to S707 in the second embodiment (FIG. 7). Steps S809 to S811 are the same as steps S708 to S710 in the second embodiment (FIG. 7).

[0071] In step S808, the system control unit 50 displays an item based on the ranking of the evaluation value of the subject, in association with the subject included in the LV image. This allows the user to direct their gaze toward the subject while taking into consideration the ranking of the evaluation value, improving convenience. For example, after the user moves their gaze away from (averts their gaze from) a favorite subject (the subject with the highest evaluation value), they can quickly return their gaze to the favorite subject. As a result, the evaluation value of the favorite subject can be maintained at number one, and the favorite subject can continue to be focused on.

[0072] Fig. 9 is a schematic diagram showing an example of the display of items. The shooting scene is a soccer scene, and LV image 900 includes three players 901 to 903 and a ball 904. In the favorite shooting mode processing of Fig. 8, evaluation values ​​are linked only to people, so evaluation values ​​are linked to players 901 to 903 but not to ball 904. In Fig. 9, items 911 to 913 and 921 to 923 are displayed superimposed on LV image 900 as items based on the ranking of evaluation values.

[0073] Items 911 to 913 are texts indicating the ranking of the evaluation value. Item 911 is displayed near contestant 901, and indicates that contestant 901's evaluation value ranking is second among contestants 901 to 903. Item 912 is displayed near contestant 902, and indicates that contestant 902's evaluation value ranking is first among contestants 901 to 903. Item 913 is displayed near contestant 903, and indicates that contestant 903's evaluation value ranking is third among contestants 901 to 903.

[0074] Items 921 to 923 are frames that surround the subject and have a pattern based on the ranking of the evaluation value of the subject. For example, the higher the ranking of the evaluation value, the closer to red (warm color) the subject is. The lower the evaluation value, the closer the frame color to blue (cooler). Item 921 surrounds the face of athlete 901 and is displayed in green. Item 922 surrounds the face of athlete 902 and is displayed in red. Item 923 surrounds the face of athlete 903 and is displayed in blue. The shape of the frame is not limited to a circle, and the ranking of the evaluation value may be indicated by the shape of the frame (such as a square or triangle), the type of line (such as a solid line or a dashed line), a flashing pattern, etc.

[0075] Although the third embodiment is based on the second embodiment, a step similar to step S808 may be added to the first embodiment (FIG. 3).

[0076] In the above description, the various controls described as being performed by the system control unit 50 may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0077] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0078] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0079] Furthermore, in the above-described embodiment, the present invention has been described as being applied to a digital camera, but this is not limited to this example and can be applied to any electronic device (imaging device) capable of capturing images. For example, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, printers, digital photo frames, music players, game consoles, e-book readers, etc. Furthermore, the present invention can be applied to video players, display devices (including projection devices), tablet terminals, smartphones, AI speakers, home appliances, in-vehicle devices, etc.

[0080] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0081] The disclosure of this embodiment includes the following configuration, method, program, and medium. (Configuration 1) A first acquisition means for acquiring line-of-sight information relating to a user's line of sight; A second acquisition means for acquiring eyeball information relating to the state of the user's eyeball; a detection means for detecting a subject; a determining means for determining an object at which the user's gaze is directed based on the gaze information and a result of detecting the object, and determining an evaluation value of the determined object based on the eyeball information; a processing means for performing autofocus processing based on the evaluation value; An imaging device comprising: (Configuration 2) The processing means performs the autofocus process so as to focus on the subject with the highest evaluation value detected by the detection means. 2. The imaging device according to claim 1, (Configuration 3) the eyeball information includes information about the pupil diameter of the user, When the user's line of sight is directed toward a subject, the determining means determines a higher evaluation value for the subject as the change in pupil diameter increases. 3. The imaging device according to configuration 1 or 2. (Configuration 4) The determining means increases the evaluation value of the subject as the time that the user directs their gaze at the subject increases. 4. The imaging device according to any one of configurations 1 to 3. (Configuration 5) The determining means repeats a process of increasing the evaluation value of the subject by an amount of change based on the eyeball information while the user is directing their gaze at the subject. 5. The imaging device according to configuration 4. (Configuration 6) The determining means reduces the evaluation value of the subject as the time during which the user does not direct his / her gaze at the subject increases. 6. The imaging device according to any one of configurations 1 to 5, wherein: (Configuration 7) The determining means repeats a process of reducing the evaluation value of the subject by an amount of change based on the eyeball information during a period in which the user is not directing their gaze at the subject. 7. The imaging device according to configuration 6, (Configuration 8) the eyeball information includes information on the user's microsaccades; The determining means Estimating the user's gaze range based on the microsaccade information; The evaluation value of the subject included in the gaze range is not reduced 8. The imaging device according to configuration 6 or 7, (Configuration 9) When a predetermined time has elapsed while the user is not directing their gaze at a predetermined type of subject, the determining means reduces the evaluation values ​​of all subjects by a predetermined reduction amount. 9. The imaging device according to any one of configurations 1 to 8. (Configuration 10) Control means for controlling the display of the captured image and The control means controls to display items based on the ranking of the evaluation value of the subject in association with the subject included in the image. 10. The imaging device according to any one of configurations 1 to 9, wherein: (Configuration 11) The control means controls to display text indicating the ranking of the evaluation value of the subject in association with the subject included in the image. 11. The imaging device according to configuration 10. (Configuration 12) The control means controls the display of a frame surrounding a subject included in the image in a manner based on the ranking of the evaluation value of the subject. 12. The imaging device according to configuration 10 or 11. (Configuration 13) In the current photographic scene, the determining means uses the evaluation value of the subject determined in the previous photographic scene as the initial value of the evaluation value of the subject. 13. The imaging device according to any one of configurations 1 to 12. (method) a first acquisition step of acquiring gaze information regarding a user's gaze; a second acquisition step of acquiring eye information relating to the state of the user's eye; a detection step of detecting an object; a determining step of determining an object at which the user's gaze is directed based on the gaze information and a result of detecting the object, and determining an evaluation value of the determined object based on the eyeball information; a processing step of performing autofocus processing based on the evaluation value; 10. A method for controlling an imaging device, comprising: (program) A program for causing a computer to function as each means of the imaging device according to any one of configurations 1 to 13. (medium) 14. A computer-readable storage medium storing a program for causing a computer to function as each means of the imaging device according to any one of configurations 1 to 13. [Explanation of symbols]

[0082] 100: Digital camera 50: System control unit

Claims

1. a first acquisition means for acquiring line-of-sight information relating to a user's line of sight; a second acquisition means for acquiring eyeball information relating to the state of the user's eyeball; a detection means for detecting a subject; a determining means for determining an object at which the user's gaze is directed based on the gaze information and a result of detecting the object, and determining an evaluation value of the determined object based on the eyeball information; a processing means for performing autofocus processing based on the evaluation value; An imaging device comprising:

2. The processing means performs the autofocus process so as to focus on the subject with the highest evaluation value detected by the detection means.

2. The imaging device according to claim 1.

3. the eyeball information includes information about the pupil diameter of the user, When the user's line of sight is directed toward a subject, the determining means determines a higher evaluation value for the subject as the change in pupil diameter increases.

2. The imaging device according to claim 1.

4. The determining means increases the evaluation value of the subject as the time that the user directs their gaze at the subject increases.

2. The imaging device according to claim 1.

5. The determining means repeats a process of increasing the evaluation value of the subject by an amount of change based on the eyeball information while the user is directing their gaze at the subject.

5. The imaging device according to claim 4.

6. The determining means reduces the evaluation value of the subject as the time during which the user does not direct his / her gaze at the subject increases.

2. The imaging device according to claim 1.

7. The determining means repeats a process of reducing the evaluation value of the subject by an amount of change based on the eyeball information during a period in which the user is not directing their gaze at the subject.

7. The imaging device according to claim 6.

8. the eyeball information includes information on the user's microsaccades; The determining means Estimating the user's gaze range based on the microsaccade information; The evaluation value of the subject included in the gaze range is not reduced 7. The imaging device according to claim 6.

9. When a predetermined time has elapsed while the user is not directing their gaze at a predetermined type of subject, the determining means reduces the evaluation values ​​of all subjects by a predetermined reduction amount.

2. The imaging device according to claim 1.

10. Control means for controlling the display of the captured image and The control means controls to display items based on the ranking of the evaluation value of the subject in association with the subject included in the image.

2. The imaging device according to claim 1.

11. The control means controls to display text indicating the ranking of the evaluation value of the subject in association with the subject included in the image.

11. The imaging device according to claim 10.

12. The control means controls the display of a frame surrounding a subject included in the image in a manner based on the ranking of the evaluation value of the subject.

11. The imaging device according to claim 10.

13. In the current photographic scene, the determining means uses the evaluation value of the subject determined in the previous photographic scene as the initial value of the evaluation value of the subject.

2. The imaging device according to claim 1.

14. a first acquisition step of acquiring gaze information relating to a user's gaze; a second acquisition step of acquiring eye information relating to the state of the user's eye; a detection step of detecting an object; a determining step of determining an object at which the user's gaze is directed based on the gaze information and a result of detecting the object, and determining an evaluation value of the determined object based on the eyeball information; a processing step of performing autofocus processing based on the evaluation value; 10. A method for controlling an imaging device, comprising:

15. A program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 13.

Images