Imaging device, control method for imaging device, and program

Abstract

An imaging device comprises a control unit for executing control using a method that corresponds to a situation where the state of a photographer determined on the basis of line-of-sight information is a state in which an object of interest is undecided. Figure 1

Description

Technical Field

[0001] The present disclosure relates to an imaging device, a control method for the imaging device, and a program.

Background Art

[0002] There is known an imaging device that performs control according to the operation of a photographer. For example, Patent Document 1 below describes a technique for monitoring whether or not a photographer is looking into a viewfinder and stopping video shooting when the photographer is not looking into the viewfinder.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is desired that appropriate control be performed not only when the photographer actively performs a predetermined operation as described in Patent Document 1 but also with respect to other states of the photographer.

[0005] One object of the present disclosure is to provide, for example, an imaging device, a control method for the imaging device, and a program that can perform appropriate control even when the state of the photographer is a state (undetermined state) in which the object of attention has not been determined.

Means for Solving the Problems

[0006] The present disclosure is, for example, Based on the line-of-sight information, determine whether the state of the photographer is a determination state in which the subject as the object of attention is determined, a non-selection state in which the object of attention is not selected, or a slow state in which the concentration of the photographer is lower than that in the non-selection state. When the state of the photographer transitions from the determination state to the non-selection state, according to the presence or absence of the subject of the object of attention in the captured image obtained in the determination state, perform the following Focus It is provided with a control unit that performs control 、 When the control unit detects that the captured image obtained in the unselected state contains the subject of interest, it performs control to focus on the subject. When the captured image obtained in the unselected state does not contain the subject of interest and contains a subject of the same type as the subject, it performs control to focus on the subject of the same type. imaging device.

[0007] Also, the present disclosure is, for example, The control unit determines, based on the line-of-sight information, whether the state of the photographer is a determination state in which the subject as the object of attention is determined, a non-selection state in which the object of attention is not selected, or a slow state in which the concentration of the photographer is lower than that in the non-selection state. When the state of the photographer transitions from the determination state to the non-selection state, perform the following Focus control Yes , When the control unit detects that the captured image obtained in the unselected state contains the subject of interest, it performs control to focus on the subject. When the captured image obtained in the unselected state does not contain the subject of interest and contains a subject of the same type as the subject, it performs control to focus on the subject of the same type. control method of the imaging device.

[0008] Also, the present disclosure is, for example, The control unit determines, based on the line-of-sight information, whether the state of the photographer is a determination state in which the subject as the object of attention is determined, a non-selection state in which the object of attention is not selected, or a slow state in which the concentration of the photographer is lower than that in the non-selection state. When the state of the photographer transitions from the determination state to the non-selection state, perform the following Focus control Yes , When the control unit detects that the captured image obtained in the unselected state contains the subject of interest, it performs control to focus on the subject. When the captured image obtained in the unselected state does not contain the subject of interest and contains a subject of the same type as the subject, it performs control to focus on the subject of the same type. A program that causes a computer to execute a method for controlling an imaging device.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure and the like will be described with reference to the drawings. The description will be made in the following order. <One Embodiment> <Modification Example> The embodiments and the like described below are preferred specific examples of the present disclosure, and the content of the present disclosure is not limited to these embodiments and the like.

[0011] <One Embodiment> [Configuration Example of Imaging Device] First, with reference to FIG. 1, a configuration example of an imaging device (imaging device 100) according to an embodiment will be described. The imaging device 100 includes a control unit 101, an optical imaging system 102, a lens driving driver 103, an imaging element 104, an image signal processing unit 105, a codec unit 106, a storage unit 107, an interface 108, an input unit 109, a display unit 110, a microphone 111, a detection unit 112, an AF control unit 113, and a gaze detection unit 114. The control unit 101 includes, as a functional block, a photographer state determination unit 101A that determines the state of the photographer based on gaze information.

[0012] The control unit 101 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The CPU executes various processes according to the programs stored in the ROM and issues commands to comprehensively control the entire imaging device 100 and each part. In addition, the control unit 101 executes control in a manner corresponding to the state of the photographer determined based on the gaze information being an undetermined state of the object of attention. Details of the processes performed by the control unit 101 will be described later.

[0013] The optical imaging system 102 is composed of an imaging lens for condensing light from a subject onto the imaging device 104, a drive mechanism for moving the imaging lens for focusing and zooming, a shutter mechanism, an iris mechanism, and the like. These are driven based on control signals from the control unit 101 and the lens drive driver 103. The optical image of the subject obtained through the optical imaging system 102 is formed on the imaging device 104.

[0014] The lens drive driver 103 is composed of, for example, a microcomputer or the like, and performs autofocus so as to focus on a target subject by moving the imaging lens along the optical axis direction by a predetermined amount based on the focus control information supplied from the AF control unit 113 or the information processing device 200. Also, in accordance with the control from the control unit 101, it controls the operations of the drive mechanism, shutter mechanism, iris mechanism, etc. of the optical imaging system 102. Thereby, adjustments such as the exposure time (shutter speed) and the aperture value (F value) are made.

[0015] The imaging device 104 photoelectrically converts the incident light from the subject obtained through the imaging lens, converts it into an electric charge amount, and outputs an imaging signal. Then, the imaging device 104 outputs a pixel signal to the image signal processing unit 105. As the imaging device 104, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), etc. are used.

[0016] The imaging device 104 has R (Red) pixels, G (Green) pixels, and B (Blue) pixels which are normal pixels, and phase difference detection pixels for performing phase difference detection. So-called phase difference AF (Auto Focus) can be performed using the phase difference information output from these phase difference detection pixels. Note that the phase difference detection pixels may function only as phase difference detection pixels and not as normal pixels, or one pixel may be composed of two independent photodiodes and function for imaging and phase difference detection. Note that the phase difference detection may be performed by an AF sensor dedicated to phase difference detection.

[0017] The image signal processing unit 105 performs sample and hold, AGC (Auto Gain Control) processing, A / D (Analog / Digital) conversion, etc. on the imaging signal output from the imaging device 104 to maintain a good S / N (Signal / Noise) ratio through CDS (Correlated Double Sampling) processing, and creates an image signal. Also, the image signal processing unit 105 performs recording processing on the image signal for recording and display processing on the image signal for display.

[0018] The codec unit 106 performs encoding processing for recording or communication, for example, on the image signal subjected to predetermined processing.

[0019] The storage unit 107 is a large-capacity storage medium such as a hard disk or a flash memory, for example. The video data and image data processed by the image signal processing unit 105 and the codec unit 106 are stored in a compressed state or an uncompressed state based on a predetermined standard. Also, EXIF (Exchangeable Image File Format) including additional information such as information about the stored data, imaging position information indicating the imaging position, and imaging time information indicating the imaging date and time is stored in association with the data.

[0020] The interface 108 is an interface with other devices or the Internet, etc. The interface 108 may include a wired or wireless communication interface. More specifically, the wired or wireless communication interface may include cellular communication such as 3TTE, Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), Ethernet (registered trademark), HDMI (registered trademark) (High-Definition Multimedia Interface), USB (Universal Serial Bus), etc.

[0021] Note that the imaging device 100 may be provided with a communication unit that can be connected to the Internet such as a wireless LAN (Local Area Network), WAN (Wide Area Network), WiFi (Wireless Fidelity), and other devices. Also, the communication between the imaging device 100 and an external device may be performed by short-range wireless communication such as NFC (Near Field Communication) or ZigBee (registered trademark), or by tethering connections such as WiFi tethering, USB (Universal Serial Bus) tethering, or Bluetooth (registered trademark) tethering.

[0022] The input unit 109 is for the photographer to give various instructions to the imaging device 100. When an input is made from the photographer to the input unit 109, a control signal corresponding to the input is generated and supplied to the control unit 101. Then, the control unit 101 performs various processes corresponding to the control signal. Examples of the input unit 109 include a shutter button for shutter input, physical buttons for various operations, a touch panel, and a touch screen integrated with the display as the display unit 110.

[0023] The display unit 110 is a display device including a display such as a through-image which is a display image signal subjected to display processing, an image / video that has been subjected to recording image processing and stored in the storage unit 107, a GUI (Graphical User Interface), etc., and an EVF (Electronic View Finder) display.

[0024] The microphone 111 is a sound collection device for recording sound during video recording.

[0025] The detection unit 112 executes AF detection for determining the focus position using the phase difference information supplied from the imaging element 104 and calculates the defocus amount. The defocus amount is supplied to the AF control unit 113.

[0026] Based on the defocus amount calculated by the detection unit 112, the AF control unit 113 generates focus control information indicating where in the angle of view (for example, XY coordinate information) to focus so as to focus on the target subject, and how much the lens drive driver 103 of the imaging device 100 moves the lens. The focus control information is information for performing autofocus control in the imaging device 100.

[0027] The line-of-sight detection unit 114 detects the line of sight of the photographer and supplies the detection result to the control unit 101.

[0028] [Regarding line-of-sight information and determination of the photographer's state] Next, a specific example of the process of detecting the line of sight of the photographer, who is the user, performed by the line-of-sight detection unit 114 will be described. The line-of-sight detection unit 114, for example, photographs the eyes of the photographer and uses the eye image to detect the line-of-sight direction of the photographer. That is, the line-of-sight detection unit 114 includes, for example, a camera unit that photographs the eyes of the photographer and a unit that detects the line-of-sight direction of the photographer. The camera unit may include a light-emitting unit that emits infrared light or the like.

[0029] As a method for detecting the line-of-sight direction of the photographer, a known method can be applied. For example, the corneal reflection method can be applied, in which infrared light or the like is irradiated from the light-emitting unit and the reflection from the cornea is used, and the line-of-sight direction of the photographer is detected based on the position of the pupil by the unit for detecting the line-of-sight direction. Also, for example, a method in which points that do not move, such as the inner corners of the eyes and the outer corners of the eyes, are recognized by image recognition and the line-of-sight direction is estimated from the position of the iris of the eye may be applied.

[0030] The line-of-sight detection unit 114 is provided, for example, inside the EVF of the imaging device 100. The line-of-sight detection unit 114 may be provided on the housing of the imaging device 100. For example, the line-of-sight detection unit 114 may be provided on the surface of the housing of the imaging device 100 where the display unit 110 is provided.

[0031] The line-of-sight direction detected by the line-of-sight detection unit 114 is supplied to the control unit 101.

[0032] The subject state determination unit 101A of the control unit 101 generates line-of-sight information from the line-of-sight detection results supplied from the line-of-sight detection unit 114. The line-of-sight information is, for example, information indicating the distribution of the line-of-sight detection results. Specifically, as shown in FIG. 2, the line-of-sight information is the distribution of the line-of-sight detection result PA at a location corresponding to the line-of-sight direction detected by the line-of-sight detection unit 114. The line-of-sight information is obtained for each appropriate region AR of the display unit 110. The region AR may be in pixel units or in block units of M pixels × N pixels (M and N are appropriate positive numbers). The subject state determination unit 101A determines the state of the subject based on the distribution of the line-of-sight detection result PA included in the line-of-sight information.

[0033] Based on the distribution of the line-of-sight detection result PA, the subject state determination unit 101A determines whether the state of the subject is a state of gazing at a subject, that is, a state of determining a subject as a gazing target (hereinafter, appropriately referred to as a gazing target determination state), or a state in which the gazing target has not yet been determined (hereinafter, appropriately referred to as a gazing target undetermined state). In the present embodiment, the subject state determination unit 101A is in a state of not selecting a gazing target, which is an example of the gazing target undetermined state (hereinafter, appropriately referred to as a gazing target unselected state), or the concentration of the subject during shooting, which is another example of the gazing target undetermined state, is lower than the Selection state (hereinafter, appropriately referred to as a slow state).

[0034] Specific examples of the state of the subject being in the gazing target determination state include a state in which the subject is shooting while following a certain subject or shooting while focusing on a desired subject. Specific examples of the state of the subject being in the gazing target unselected state include a state in which the subject is hesitating about which of a certain subject and another subject to focus on for shooting. Specific examples of the state of the subject being in the slow state include the state of a subject who is observing a bird's nest while looking through a viewfinder and waiting for the birds to return to the nest, and the state of the subject during the halftime in soccer shooting or during the scene of the change of offense and defense in baseball shooting.

[0035] The photographer state determination unit 101A determines, for example, based on the distribution of the gaze detection results, whether the state of the photographer is any of the above-described three states. For example, as shown in FIG. 2, the photographer state determination unit 101A obtains a histogram of the values of the distribution of the gaze detection result PA corresponding to both the X-axis direction and the Y-axis direction in the display unit 110, and obtains the variance corresponding to each peak. Then, when there is a peak with a variance smaller than a predetermined value, the photographer state determination unit 101A determines that the state of the photographer is the determined state of the target of fixation.

[0036] Also, when the distribution of the gaze detection results does not have a peak with a variance value smaller than a predetermined value, the photographer state determination unit 101A determines that the state of the photographer is the undetermined state of the target of fixation. More specifically, when the distribution of the gaze detection results indicated by the gaze information has only a plurality of peaks with a variance value larger than a predetermined value, that is, when the variance values of all the peaks are larger than a predetermined value, the state of the photographer is the unselected state of the target of fixation, and when the distribution of the gaze detection results has a variance value larger than a predetermined value, the photographer state determination unit 101A determines that the state of the photographer is the relaxed state.

[0037] Note that the photographer state determination unit 101A may determine the state of the photographer based on the locus of the user's gaze detected by the gaze detection unit 114. For example, when the locus of the user's gaze within a predetermined time is within a certain range, the photographer state determination unit 101A determines that the state of the photographer is the determined state of the target of fixation. Also, when the locus of the user's gaze within a predetermined time reciprocates at a plurality of locations (for example, two locations, or three or more locations) of the display image, the photographer state determination unit 101A determines that the state of the photographer is the undetermined state of the target of fixation. Further, when the locus of the user's gaze within a predetermined time is irregularly distributed, the photographer state determination unit 101A determines that the state of the user is the relaxed state. In other words, the relaxed state is the state of the photographer corresponding to the fact that the locus of the gaze detection result indicated by the gaze information is wide compared to the determined state of the target of fixation.

[0038] In addition, the photographer state determination unit 101A may determine the state of the photographer based on the number of peaks having a predetermined sharpness or more that appear when the histogram of the distribution of the gaze detection results is smoothed. For example, if the number of peaks having sharpness is 1, the photographer state determination unit 101A determines that the state of the photographer is the determined state of the target of attention. If the number of peaks having sharpness is within the range of 2 to 4, the state of the photographer is the unselected state of the target of attention. If the number of peaks having sharpness is 5 or more, the state of the photographer is determined to be a slow state.

[0039] [Flow of processing] (Overall flow of processing) Next, with reference to the flowchart of FIG. 3, the flow of processing (overall processing) performed by the imaging device 100 will be described. The processing described below is performed, for example, when the power of the imaging device 100 is turned on or when the shooting mode is switched. The processing may be started at the stage when the user assumes a posture in which the imaging device 100 will be lifted and photographed according to the detection results of the gyro sensor and the pressure sensor. Alternatively, the processing may be started after the gaze is detected by the gaze detection unit 114. The processing may be repeated while the gaze is detected. In addition, the processing described below is performed by the control unit 101 unless otherwise specified. The same applies to the description of the flow of other processing.

[0040] In step ST11, the photographer state determination unit 101A detects (generates) gaze information based on the gaze detection result by the gaze detection unit 114. Then, the processing proceeds to step ST12.

[0041] In step ST12, the photographer state determination unit 101A determines the state of the photographer based on the gaze information. If the state of the photographer is the determined state of the target of attention, the processing proceeds to step ST13. If the state of the photographer is the unselected state of the target of attention, the processing proceeds to step ST14. If the state of the photographer is the slow state, the processing proceeds to step ST15.

[0042] In step ST13, the control unit 101 performs control in a manner corresponding to the state of the photographer, that is, the determination state of the object of attention. Such control is control for providing assistance according to the state of the photographer. As a specific example, there is control for displaying an AF pointer and aligning the AF pointer with the subject that is the object of attention or performing AF on the subject that is the object of attention.

[0043] In step ST14, the control unit 101 performs control in a manner corresponding to the state of the photographer, that is, the non-selection state of the object of attention. In step ST15, the control unit 101 performs control in a manner corresponding to the state of the photographer, that is, the slow state of the object of attention.

[0044] (Specific example of control corresponding to the non-selection state of the object of attention) Next, a specific example of control corresponding to the non-selection state of the object of attention will be described. When the state of the photographer is the non-selection state of the object of attention, the control unit 101 determines a focus area so as to include the target subject. Note that including the target subject means including at least a part of the target subject. Also, determining the focus area means determining a part of the area in the display unit 110 as the focus area. Also, the subject is a subject recognized by known image recognition processing. The control unit 101 determines the target subject based on, for example, the shooting situation before the state of the photographer becomes the non-selection state of the object of attention. As an example, the control unit 101 determines the target subject based on the subject that was the object of attention before the state of the photographer becomes the non-selection state of the object of attention. Note that the subject that occupies the largest area or has the largest movement in the display image currently displayed on the display unit 110 may be determined as the target subject.

[0045] A specific example will be described with reference to FIG. 4. FIG. 4A is a diagram showing an example of a captured image (an example of a display image displayed on the display unit 110) obtained in a state where the state of the photographer is a state of determining a target of fixation. As shown in FIG. 4A, the photographer is gazing at the batter BH as a subject to be gazed at. The AF control unit 113 is controlled so that the face of the batter BH is in focus. An AF frame 21 and an AF pointer 22 are displayed on the face of the batter BH.

[0046] Here, it is assumed that the captured image is changed from the one shown in FIG. 4A to the one shown in FIG. 4B by changing the angle of view. In this case, the display position of the batter BH, which is the target of AF, changes as the angle of view changes. In the captured image shown in FIG. 4B, the photographer is in a state of being confused about which subject to focus on, that is, the state of the photographer becomes a state of non-selection of the target of fixation. Usually, since it is assumed that the photographer wants to continue focusing on the subject that has been gazed at so far (the face of the batter BH in this example), the control unit 101 controls the AF control unit 113 so that the face of the batter BH is in focus. In response to such control, as shown in FIG. 4C, the AF frame 21 and the AF pointer 22 are displayed on the face of the batter BH.

[0047] Next, an example of the flow of control (processing) corresponding to the state of non-selection of the target of fixation will be described with reference to the flowchart of FIG. 5. The processing described below is performed, for example, when the photographer state determination unit 101A determines that the state of the photographer is a state of non-selection of the target of fixation. Also, the processing described below may be repeated while the line of sight is detected.

[0048] In step ST21, the control unit 101 starts control in a manner corresponding to the case where the state of the photographer is a state of non-selection of the target of fixation. Then, the process proceeds to step ST22.

[0049] In step ST22, it is determined whether or not the subject (the target subject in this example) that was in focus at the time of the fixation target determination state is included in the current captured image. If the subject that was in focus is included (Yes), the process proceeds to step ST23.

[0050] In step ST23, control is performed to focus on the subject that was in focus at the time of the fixation target determination state. The process from step ST22 to step ST23 is the process described with reference to FIGS. 4A to 4C described above.

[0051] In the determination process of step ST22, if the subject that was in focus at the time of the fixation target determination state is not included in the current captured image (No), the process proceeds to step ST24.

[0052] In step ST24, it is determined whether or not the current captured image includes a subject of the same type as the subject that was in focus at the time of the fixation target determination state by performing subject recognition processing or the like. If a subject of the same type is included (Yes), the process proceeds to step ST25.

[0053] In step ST25, a subject of the same type as the subject that was in focus at the time of the fixation target determination state is determined as the target subject, and control is performed to focus on the subject.

[0054] In the determination process of step ST24, if the current captured image does not include a subject of the same type as the subject that was in focus at the time of the fixation target determination state (No), the process proceeds to step ST26.

[0055] In step ST26, among the subjects currently displayed on the display unit 110, the subject with the largest area is determined as the target subject, and control is performed to focus on the subject.

[0056] As a specific example of the process according to step ST24, for example, in the case of taking a picture of an image in which multiple horses and riders are mixed in horse racing, when the state of the photographer is in an unselected state of the target of attention, it is determined whether the type in focus before that (for example, the type of animal including humans, the human attribute types such as gender and race, the types of objects such as desks, houses, balls, etc.) is included in the current captured image. If a person is included, control is performed to focus on the person, and if no person is included, control is performed to focus on the subject with a large area (for example, a horse).

[0057] In addition, when the state of the photographer is in an unselected state of the target of attention, the control unit 101 may perform highlighting on the target subject. Examples of highlighting include increasing the brightness of the target subject, decreasing the brightness around the target subject, blinking the target subject, and displaying a mark on the target subject or in its vicinity.

[0058] (Specific examples of control corresponding to the slow state) Next, specific examples of control corresponding to the unselected state of the target of attention will be described. When it is determined that the state of the photographer is in a slow state, for example, control is performed to put the imaging device 100 into a power-saving mode by stopping a part of the functions or reducing the performance. For example, the display frame rate of the EVF display is reduced to switch the imaging device 100 to the power-saving mode. Also, when the state of the photographer is in a slow state, a scene where the photographer is searching for a subject in sports photography with many athletes is also assumed. Therefore, in order to make it easier to capture the subject, control is performed to widen the ranging range and automatically widen the focus area (focus wide mode). Also, when the state of the photographer changes from a state other than the slow state (for example, the determined state of the target of attention or the unselected state of the target of attention) to the slow state, the angle of view may be made wider than the angle of view when not in the slow state. In addition, when the state of the photographer is in a slow state with low concentration, since there is a risk of missing an important scene when in the slow state, control (image saving control) is performed to automatically save the images continuously captured by shooting to the buffer memory. Hereinafter, the control corresponding to the slow state will be described more specifically.

[0059] (First example) FIG. 6 is a flowchart showing a first example of a control flow corresponding to a slow state. The first example is an example in which the control unit 101 selects a control method corresponding to the shooting situation from among a plurality of control methods when the state of the photographer is in the slow state. More specifically, it is an example of selecting a control method corresponding to the shooting mode as the shooting situation from among a plurality of control methods. Note that the processes described below may be repeated while the line of sight is detected.

[0060] In step ST31, the photographer state determination unit 101A determines that the state of the photographer is in the slow state, and the control unit 101 starts control in a manner corresponding to the slow state. Then, the process proceeds to step ST32.

[0061] In step ST32, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Note that it is not always necessary to set the focus wide mode (the same applies to other examples). Then, the process proceeds to step ST33.

[0062] In step ST33, the shooting mode of the imaging device 100 is determined. Here, when the shooting mode is the program auto mode (hereinafter, appropriately referred to as the P mode), the process proceeds to step ST34. When the shooting mode is the shutter speed priority mode (hereinafter, appropriately referred to as the S mode), the process proceeds to step ST35. When the shooting mode is other modes (for example, the auto mode), the process proceeds to step ST36.

[0063] In step ST34, control is performed to set the angle-of-view wide mode to widen the angle of view so that shooting is performed with a wider angle of view. Since the P mode is generally set when the photographer is a beginner or an amateur, the angle-of-view wide mode is set to assist such a photographer.

[0064] In step ST35, control is performed to set a storage mode for storing images continuously captured by shooting in a buffer memory (e.g., a ring buffer). In the case of the S mode, since a scene of shooting a subject with relatively fast movement is assumed, the storage mode is set to avoid missing a decisive scene. The buffer length of the ring buffer may be increased according to the time of the slow state.

[0065] In step ST36, control is performed to set a power saving mode for reducing the power consumption in the imaging device 100. Specific examples of the power saving mode include, in addition to the above-described control, controlling to limit the output band from the imaging element 104, controlling to lower the luminance of the display unit 110, and the like.

[0066] (Second example) FIG. 7 is a flowchart showing a second example of the control flow corresponding to the slow state. The second example is an example in which the control unit 101 selects a control method corresponding to the shooting situation from among a plurality of control methods when the state of the photographer is the slow state. More specifically, it is an example of selecting a control method corresponding to the type of subject as the shooting situation from among a plurality of control methods. Note that the processing described below may be repeated while the line of sight is detected.

[0067] In step ST41, the shooting state determination unit 101A determines that the state of the photographer is the slow state, and the control unit 101 starts control in a manner corresponding to the slow state. Then, the process proceeds to step ST42.

[0068] In step ST42, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Then, the process proceeds to step ST43.

[0069] In step ST43, a subject is determined by performing subject recognition processing. If the subject includes something with movement (for example, a person, an animal, a vehicle, etc.), the process proceeds to step ST44. If the subject does not include something with movement (for example, only a landscape), the process proceeds to step ST45.

[0070] In step ST44, since the subject includes something with movement, control is performed to set the save mode so as not to miss a decisive scene. On the other hand, in step ST45, since the subject does not include something with movement, control is performed to set the power saving mode.

[0071] (The third example) FIG. 8 is a flowchart showing a third example of the control flow corresponding to the slow state. The third example is an example in which the control unit 101 selects a control method corresponding to the duration of the slow state from among a plurality of control methods when the state of the photographer is the slow state. Note that the processing described below may be repeated while the line of sight is detected.

[0072] In step ST51, the photographer state determination unit 101A determines that the state of the photographer is the slow state, and the control unit 101 starts control in a manner corresponding to the slow state. Then, the process proceeds to step ST52.

[0073] In step ST52, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Then, the process proceeds to step ST53.

[0074] In step ST53, it is determined whether or not the state in which the state of the photographer is the slow state has continued for a predetermined time or more (for example, 1 minute to several minutes). If it has continued (Yes), the process proceeds to step ST54. If it has not continued (No), the process proceeds to step ST55.

[0075] In step ST54, control is performed to set the imaging device 100 to the save mode. In step ST55, control is performed to set the imaging device 100 to the power saving mode.

[0076] As a specific situation of this example, for example, photographing birds or wild animals is assumed. In the case of a professional photographer lying in wait for an animal, it is detected that there is a line of sight while looking through the viewfinder, but the line of sight is not concentrated, that is, it is in a relaxed state. When a certain amount of time (for example, 1 minute or more) has passed in the relaxed state, control is performed to set the save mode assuming that the photographer is waiting for an animal whose arrival time is unknown. On the other hand, when the relaxed state does not continue, control is performed to set the power saving mode assuming that the photographer is not waiting.

[0077] (Fourth example) FIG. 9 is a flowchart showing a fourth example of the control flow corresponding to the relaxed state. The fourth example is an example in which the control unit 101 selects a control method corresponding to the shooting situation from among a plurality of control methods when the state of the photographer is in the relaxed state. More specifically, it is an example of selecting a control method corresponding to the change in the captured image as the shooting situation from among a plurality of control methods. Note that the processing described below may be repeated while the line of sight is detected.

[0078] In step ST61, the photographer state determination unit 101A determines that the state of the photographer is in the relaxed state, and the control unit 101 starts control in a manner corresponding to the relaxed state. Then, the process proceeds to step ST62.

[0079] In step ST62, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Then, the process proceeds to step ST63.

[0080] In step ST63, it is determined whether the change in the image is within the threshold. The change in the image is determined based on, for example, the correlation between the current frame and a past frame (e.g., the frame one frame before in time). If the correlation is equal to or greater than a certain value, it is determined that the change in the image is within the threshold, and if the correlation is less than the certain value, it is determined that the change in the image is greater than the threshold. If the change in the image is within the threshold (Yes case), the process proceeds to step ST64. Also, if the change in the image is greater than the threshold, the process proceeds to step ST65.

[0081] In step ST64, control is performed to set the saving mode in the imaging device 100. In step ST65, control is performed to set the power saving mode in the imaging device 100.

[0082] As a specific situation in this example, similar to the third example described above, for example, photographing birds or wild animals is assumed. In the case of a professional photographer lying in wait for an animal, it is detected that there is a line of sight while looking through the viewfinder, but the line of sight is not concentrated, that is, it is in a relaxed state. Under such circumstances, generally, the scenery during shooting is mostly shown and the change in the image is small. Therefore, when the change in the image is small, in order not to miss a decisive scene (e.g., a scene in which an animal is captured), control is performed to set the saving mode. On the other hand, when the change in the image is large, it is determined that it is not the above-described situation, and control is performed to set the power saving mode.

[0083] According to the present embodiment described above, not only when the photographer actively performs a predetermined operation, but also when the state of other photographers, specifically, the state of the photographer is in a state of unselected target of attention or a relaxed state, appropriate control can be performed.

[0084] <Modification Example> As described above, one embodiment of the present disclosure has been specifically described. However, the content of the present disclosure is not limited to the above-described embodiment, and various modifications based on the technical idea of the present disclosure are possible.

[0085] In the above-described embodiment, there may be a case where control is not performed in a manner corresponding to the state of the photographer determined based on the line-of-sight information. FIG. 10 is a flowchart showing the flow of processing performed in such a case. Note that descriptions of content overlapping with that described in FIG. 3 will be omitted as appropriate.

[0086] After the processing of step ST11, the processing according to step ST71 is performed. In step ST71, it is determined whether the photographer is in the middle of an operation for setting the imaging device 100. Examples of the setting operation include operations for setting the shutter speed and ISO sensitivity. Whether the photographer is in the middle of an operation for setting the imaging device 100 can be determined based on the presence or absence of an operation signal generated according to the operation, or based on the sensing result by a sensor provided in the input unit 109 such as a dial.

[0087] If it is determined in step ST71 that the operation is in progress (Yes), the process proceeds to step ST72 and a series of processes ends. If it is determined in step ST71 that the operation is not in progress (No), the process proceeds to step ST12. Since the processing after step ST12 has already been described, duplicate descriptions will be omitted.

[0088] When the photographer is performing an active setting operation, if control is performed according to the state of the photographer and settings are made according to such control, there is a possibility that the settings will be different from the settings intended by the photographer. In this example, since it is determined whether the operation related to the setting is in progress, the above-mentioned inconvenience can be avoided.

[0089] FIG. 11 is a flowchart showing the flow of processing performed in another case where control is not performed according to the state of the photographer determined based on the line-of-sight information. After the processing of step ST11, the processing according to step ST81 is performed.

[0090] In step ST81, it is determined whether the mode of the imaging device 100 is the manual mode (hereinafter, appropriately referred to as the M mode). When the mode of the imaging device 100 is the M mode (Yes), the process proceeds to step ST82 and a series of processes ends. When the mode of the imaging device 100 is not the M mode (No), the process proceeds to step ST12. Since the processes after step ST12 have already been described, duplicate descriptions are omitted.

[0091] There may be cases where, when the photographer is an advanced user, it is not desired to perform control in a manner according to the state of the photographer. In this case, the photographer can avoid the above-mentioned inconveniences by setting the mode of the imaging device 100 to the M mode. Note that not limited to the M mode, a dedicated mode in which control according to the state of the photographer is not performed may be settable.

[0092] In the above-described embodiment, a temporal element may be considered when determining the state of the photographer. For example, even when it is determined based on the gaze information that the state of the photographer is a relaxed state, if that state continues for a predetermined time (for example, 5 seconds) or more, control according to the relaxed state may be started.

[0093] Note that in the above-described embodiment, the photographer state determination unit 101A may determine that the state of the photographer is any one state.

[0094] In the process according to the above-described embodiment, a process based on machine learning, that is, a process using a learning model obtained by learning performed in advance may be performed.

[0095] The configurations, methods, processes, shapes, materials, numerical values, etc. described in the above embodiments and modified examples are merely examples, and different configurations, methods, processes, shapes, materials, numerical values, etc. may be used as necessary, and it is also possible to replace them with known ones. Further, the configurations, methods, processes, shapes, materials, numerical values, etc. in the embodiments and modified examples can be combined with each other as long as no technical contradiction occurs.

[0096] Note that the content of the present disclosure is not limitedly interpreted by the effects exemplified in this specification.

[0097] The present disclosure can also adopt the following configurations. (1) An imaging device including a control unit that executes control in a manner corresponding to the state of the photographer determined based on the line-of-sight information being an undetermined state of the object of interest. Imaging device. (2) The undetermined state of the object of interest is a state of the photographer corresponding to the distribution of the line-of-sight detection results indicated by the line-of-sight information not having a peak of a dispersion value smaller than a predetermined value. The imaging device according to (1). (3) The control unit determines the state of the photographer based on the line-of-sight information. The imaging device according to (1) or (2). (4) When the state of the photographer is the unselected state of the object of interest which is the undetermined state of the object of interest, the control unit determines a focus area to include the target subject. The imaging device according to any one of (1) to (3). (5) When the state of the photographer is the unselected state of the object of interest, the control unit determines the target subject based on the shooting situation before the state of the photographer becomes the unselected state of the object of interest. The imaging device according to (4). (6) The control unit determines the target subject based on the subject that was the object of fixation before the state of the photographer becomes a state where the object of fixation is not selected. The imaging device according to (5). (7) The state where the object of fixation is not selected corresponds to the state of the photographer in which the distribution of the line-of-sight detection results indicated by the line-of-sight information has only a plurality of peaks having a dispersion value larger than a predetermined value. The imaging device according to any one of (4) to (6). (8) The state where the object of fixation is not selected corresponds to the state of the photographer in which the locus of the line-of-sight detection results indicated by the line-of-sight information reciprocates at a plurality of locations on the display image. The imaging device according to any one of (4) to (6). (9) When the state of the photographer is a slow state where the state is undetermined, the control unit widens the angle of view when changing from a state that is not the slow state to the slow state. The imaging device according to any one of (1) to (8). (10) When the state of the photographer is a slow state where the state is undetermined, the control unit performs power saving control. The imaging device according to any one of (1) to (8). (11) When the state of the photographer is a slow state where the state is undetermined, the control unit performs image saving control. The imaging device according to any one of (1) to (8). (12) When the state of the photographer is a slow state where the state is undetermined, the control unit widens the distance measurement range when changing from a state that is not the slow state to the slow state. The imaging device according to any one of (1) to (8). (13) When the state of the photographer is a slow state, the control unit selects a control method corresponding to the shooting situation from among a plurality of control methods. The imaging device according to any one of (1) to (8). (14) When the state of the photographer is the slow state, the control unit selects the control method corresponding to the shooting mode as the shooting situation from among the plurality of control methods. The imaging device according to (13). (15) When the state of the photographer is the slow state, the control unit selects the control method corresponding to the type of subject as the shooting situation from among the plurality of control methods. The imaging device according to (13). (16) When the state of the photographer is the slow state, the control unit selects the control method corresponding to the duration of the slow state as the shooting situation from among the plurality of control methods. The imaging device according to (13). (17) The slow state is a state of the photographer corresponding to the distribution of the line-of-sight detection result indicated by the line-of-sight information having a variance value larger than a predetermined value. The imaging device according to any one of (9) to (16). (18) The slow state is a state of the photographer corresponding to the locus of the line-of-sight detection result indicated by the line-of-sight information being wide compared to the determination state of the fixation target. The imaging device according to any one of (9) to (16). (19) The control unit executes control in a manner corresponding to the state of the photographer determined based on the line-of-sight information being the undetermined state of the fixation target. A control method for an imaging device. (20) The control unit executes control in a manner corresponding to the state of the photographer determined based on the line-of-sight information being the undetermined state of the fixation target. A program for causing a computer to execute a control method for an imaging device.

Explanation of Signs

[0098] 100 ··· Imaging device 101 ··· Control unit 101A ··· Photographer state determination unit 114 ··· Gaze detection unit

Claims

1. Based on the line-of-sight information, it is determined whether the state of the photographer is a determination state in which the subject as the object of attention is determined, a non-selection state in which the object of attention is not selected, or a slow state in which the concentration of the photographer is lower than that in the non-selection state. When the state of the photographer transitions from the determination state to the non-selection state, a control unit is provided that performs focus control on the captured image obtained in the non-selection state according to the presence or absence of the subject that is the object of attention in the captured image obtained in the determination state. The control unit: When the captured image obtained in the non-selection state includes the subject that is the object of attention, performs control to focus on the subject. When the captured image obtained in the non-selection state does not include the subject that is the object of attention and the captured image obtained in the non-selection state includes a subject of the same type as the subject, performs control to focus on the subject of the same type. An imaging device.

2. The non-selection state and the slow state are the states of the photographer corresponding to the fact that the distribution of the line-of-sight detection results indicated by the line-of-sight information does not have a peak with a dispersion value smaller than a predetermined value. The imaging device according to claim 1.

3. The non-selection state of the object of attention is the state of the photographer corresponding to the fact that the locus of the line-of-sight detection results indicated by the line-of-sight information reciprocates at a plurality of locations on the display image. The imaging device according to claim 1.

4. When the state of the photographer is the slow state, the control unit widens the angle of view when the state changes from a non-slow state to the slow state. The imaging device according to claim 1.

5. When the state of the photographer is the slow state, the control unit performs power-saving control. The imaging device according to claim 1.

6. When the state of the photographer is the slow state, the control unit performs image storage control. The imaging device according to claim 1.

7. When the state of the photographer is the slow state, the control unit widens the ranging range when the state changes from a non-slow state to the slow state. The imaging device according to claim 1.

8. When the state of the photographer is the slow state, the control unit selects a control method corresponding to the shooting situation from among a plurality of control methods. The imaging device according to claim 1.

9. When the state of the photographer is the slow state, the control unit selects, from among the plurality of control methods, the control method corresponding to the shooting mode as the shooting situation. The imaging device according to claim 8.

10. When the state of the photographer is the slow state, the control unit selects, from among the plurality of control methods, the control method corresponding to the type of subject as the shooting situation. The imaging device according to claim 9.

11. When the state of the photographer is the slow state, the control unit selects, from among the plurality of control methods, the control method corresponding to the duration of the slow state as the shooting situation. The imaging device according to claim 9.

12. The slow state is a state of the photographer corresponding to the distribution of the gaze detection result indicated by the gaze information having a variance value greater than a predetermined value. The imaging device according to claim 1.

13. The slow state is a state of the photographer corresponding to the locus of the gaze detection result indicated by the gaze information being wide compared to the determination state. The imaging device according to claim 1.

14. The control unit determines, based on the gaze information, whether the state of the photographer is a determination state in which the subject as the gaze target has been determined, a non-selection state in which the gaze target has not been selected, or a slow state in which the concentration of the photographer is lower than that in the non-selection state. When the state of the photographer transitions from the determination state to the non-selection state, focus control is performed on the captured image obtained in the non-selection state according to the presence or absence of the subject of the gaze target in the captured image obtained in the determination state. The control unit When the captured image obtained in the non-selection state includes the subject of the gaze target, control is performed to focus on the subject. When the captured image obtained in the non-selection state does not include the subject of the gaze target and the captured image obtained in the non-selection state includes a subject of the same type as the subject, control is performed to focus on the subject of the same type. A control method for an imaging device.

15. The control unit determines, based on the line-of-sight information, whether the state of the photographer is a determination state in which the subject as the object of attention has been determined, a non-selection state in which the object of attention has not been selected, or a slow state in which the concentration of the photographer is lower than that in the non-selection state. When the state of the photographer transitions from the determination state to the non-selection state, focus control corresponding to the presence or absence of the subject that is the object of attention in the captured image obtained in the determination state is performed on the captured image obtained in the non-selection state. The control unit is configured to: When the captured image obtained in the non-selection state includes the subject that is the object of attention, perform control to focus on the subject. When the captured image obtained in the non-selection state does not include the subject that is the object of attention and includes a subject of the same type as the subject in the captured image obtained in the non-selection state, perform control to focus on the subject of the same type. A program for causing a computer to execute a control method of an imaging device.

Images