Imaging apparatus, method of controlling imaging apparatus, and program product

Abstract

An imaging apparatus includes a control unit that performs control in a manner corresponding to the fact that a state of an image capturer determined on the basis of line-of-sight information is a gaze-target-undetermined state in which a gaze target has not been determined.

Description

Technical Field

[0001] The present disclosure relates to an imaging device, a control method of the imaging device, and a program. Background Art

[0002] An imaging device that performs control according to an operation of an image capturer is known. For example, Patent Document 1 below describes a technique for monitoring whether an image capturer is looking at a viewfinder and stopping the capture of a moving image when the image capturer is not looking at the viewfinder.

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2008-306533 Summary of the Invention

[0006] Problems to be solved by the present invention

[0007] As described in Patent Document 1, it is desirable to appropriately perform control not only on a state in which an image capturer actively performs a predetermined operation but also on other states of the image capturer.

[0008] An object of the present disclosure is to provide, for example, an imaging device, a control method of the imaging device, and a program capable of performing appropriate control even in a state (undetermined state) in which an image capturer has not determined a gaze target.

[0009] Solution to the problem

[0010] The present disclosure provides, for example,

[0011] An imaging device comprising

[0012] A control unit that performs control in a manner corresponding to the fact that the state of the image capturer determined based on the line of sight information is a gaze target undetermined state in which a gaze target has not yet been determined.

[0013] Furthermore, the present disclosure provides, for example,

[0014] A method for controlling an imaging device, the method comprising:

[0015] The control unit is caused to perform control in a manner corresponding to the fact that the state of the image capturer determined based on the visual line information is a gaze target undetermined state in which a gaze target has not yet been determined.

[0016] Furthermore, the present disclosure provides, for example,

[0017] A program for causing a computer to execute a method of controlling an imaging device, the method comprising

[0018] The control unit is caused to perform control in a manner corresponding to the fact that the state of the image capturer determined based on the visual line information is a gaze target undetermined state in which a gaze target has not yet been determined. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram for explaining a configuration example of an imaging device according to an embodiment.

[0020] Figure 2 is a diagram illustrating an example of the distribution of gaze detection results.

[0021] Figure 3 is a flowchart illustrating the flow of processing performed by the imaging device according to the embodiment.

[0022] Figure 4 4A to 4C are diagrams for explaining an example of control performed in a case where the state of the image capturer is a gaze target unselected state.

[0023] Figure 5 is a diagram for explaining an example of the flow of control performed in a case where the state of the image capturer is a gaze target unselected state.

[0024] Figure 6 is a diagram for explaining an example of the flow of control performed in a case where the state of the image capturer is an idle state.

[0025] Figure 7 is a diagram for explaining another example of the flow of control performed in a case where the state of the image capturer is the idle state.

[0026] Figure 8 is a diagram for explaining another example of the flow of control performed in a case where the state of the image capturer is the idle state.

[0027] Figure 9 is a diagram for explaining another example of the flow of control performed in a case where the state of the image capturer is the idle state.

[0028] Figure 10 is a flowchart used to explain the modification.

[0029] Figure 11 is a flowchart used to explain the modification. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present disclosure and the like will be described with reference to the accompanying drawings. Note that the description will be given in the following order.

[0031] <One embodiment>

[0032] <Edit>

[0033] The embodiments and the like described below are preferred specific examples of the present disclosure, and the contents of the present disclosure are not limited to the embodiments and the like.

[0034] <One embodiment>

[0035] [Configuration Example of Imaging Device]

[0036] First, refer to Figure 1 An example configuration of an imaging device (imaging device 100) according to an embodiment will be described. The imaging device 100 includes a control unit 101, an optical imaging system 102, a lens drive driver 103, an imaging element 104, an image signal processing unit 105, a codec unit 106, a storage unit 107, an interface 108, an input unit 109, a display unit 110, a microphone 111, a detection unit 112, an AF control unit 113, and a line of sight detection unit 114. The control unit 101 includes an image capturer state determination unit 101A as a functional block that determines the state of the image capturer based on line of sight information.

[0037] The control unit 101 includes a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), and the like. The CPU executes various processes and issues commands according to the programs stored in the ROM, thereby comprehensively controlling the entire imaging device 100 and each unit. Furthermore, the control unit 101 performs control in a manner corresponding to the fact that the state of the image capturer, determined based on the line of sight information, is a gaze target undetermined state, where the gaze target is undetermined. Details of the processing performed by the control unit 101 will be described later.

[0038] The optical imaging system 102 includes an imaging lens for focusing light from a subject on the imaging element 104, a drive mechanism for moving the imaging lens to perform focusing and zooming, a shutter mechanism, an aperture mechanism, and the like. These are driven based on control signals from the control unit 101 and the lens drive driver 103. An optical image of the subject obtained by the optical imaging system 102 is formed on the imaging element 104.

[0039] The lens drive driver 103 includes, for example, a microcomputer and moves the imaging lens by a predetermined amount along the optical axis direction based on focus control information supplied from the AF control unit 113 or the information processing device 200, thereby performing autofocus to focus on the target subject. Furthermore, the operation of the drive mechanism, shutter mechanism, aperture mechanism, and the like of the optical imaging system 102 is controlled under the control of the control unit 101. Thus, the exposure time (shutter speed) and the aperture value (F value) are adjusted.

[0040] The imaging element 104 photoelectrically converts light incident from a subject and obtained through an imaging lens into an electric charge and outputs an imaging signal. The imaging element 104 then outputs the pixel signal to the image signal processing unit 105. As the imaging element 104, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like is used.

[0041] The imaging element 104 includes red (R), green (G), and blue (B) pixels as normal pixels, as well as phase difference detection pixels that perform phase difference detection. It is possible to use the phase difference information output from the phase difference detection pixels to perform so-called phase difference autofocus (AF). Note that the phase difference detection pixels may function only as phase difference detection pixels and may not function as normal pixels, or may be configured as one pixel by two independent photodiodes to perform imaging and phase difference detection. Note that phase difference detection may be performed by an AF sensor dedicated to phase difference detection.

[0042] The image signal processing unit 105 creates an image signal by performing sampling and holding to maintain a high signal-to-noise ratio (S / N) by performing correlated double sampling (CDS) processing, automatic gain control (AGC) processing, analog / digital (A / D) conversion, etc. on the imaging signal output from the imaging element 104. In addition, the image signal processing unit 105 performs recording processing on the image signal for recording and performs display processing on the image signal for display.

[0043] The codec unit 106 performs encoding processing for recording and communication, for example, on the image signal subjected to predetermined processing.

[0044] The storage unit 107 is a large-capacity storage medium such as a hard disk or a flash memory. The video data and image data processed by the image signal processing unit 105 and the codec unit 106 are stored in a compressed or uncompressed state based on a predetermined standard. In addition, an Exchangeable Image File Format (EXIF) file containing additional information (such as information about the stored data, imaging position information indicating the imaging position, and imaging time information indicating the imaging date and time) is also stored in association with the data.

[0045] The interface 108 is an interface with another device, the Internet, or the like. The interface 108 may include a wired or wireless communication interface. Specifically, the wired or wireless communication interface may include cellular communication such as 3TTE, Wi-Fi, Bluetooth (registered trademark), near field communication (NFC), Ethernet (registered trademark), High Definition Multimedia Interface (HDMI) (registered trademark), Universal Serial Bus (USB), or the like.

[0046] Note that the imaging device 100 may include a communication unit such as a wireless local area network (LAN), a wide area network (WAN), or wireless fidelity (WiFi) that can connect to the Internet, other devices, etc. In addition, the communication between the imaging device 100 and the external device may be a short-range wireless communication such as near field communication (NFC) or ZigBee (registered trademark), or a tethered connection such as WiFi tethering, universal serial bus (USB) tethering, or Bluetooth (registered trademark) tethering.

[0047] The image capturer uses the input unit 109 to give various instructions to the imaging device 100. When the image capturer inputs the input unit 109, a control signal corresponding to the input is generated and supplied to the control unit 101. The control unit 101 then performs various processes corresponding to the control signal. Examples of the input unit 109 include a shutter button for shutter input, physical buttons for various operations, a touch panel, a touch screen integrated with a display as the display unit 110, and the like.

[0048] The display unit 110 is a display device that includes a display that displays a through image as a display image signal that has undergone display processing, an image / video that has undergone recording image processing and is stored in the storage unit 107, a graphical user interface (GUI), an electronic viewfinder (EVF) display, etc.

[0049] The microphone 111 is a sound collecting device for recording sound at the time of recording.

[0050] The detection unit 112 performs AF detection for determining a focus position and calculates a defocus amount using the phase difference information supplied from the imaging element 104. The defocus amount is supplied to the AF control unit 113.

[0051] The AF control unit 113 generates focus control information indicating which position in the angle of view to focus on (e.g., XY coordinate information) and how much the lens drive driver 103 of the imaging device 100 should move the lens to focus on the target subject based on the defocus amount calculated by the detection unit 112. The focus control information is information used to perform automatic focus control in the imaging device 100.

[0052] The sight line detection unit 114 detects the sight line of the image capturer and supplies the detection result to the control unit 101 .

[0053] [Regarding determination of line of sight information and state of image capturer]

[0054] Next, we will describe a specific example of the process by which gaze detection unit 114 detects the gaze of an image capturer (a user). For example, gaze detection unit 114 captures an image of the image capturer's eyes and uses the eye image to detect the direction of the image capturer's gaze. Specifically, gaze detection unit 114 includes, for example, a camera unit that captures the captured person's eyes and a unit that detects the direction of the image capturer's gaze. The camera unit may include a light-emitting unit that emits infrared light or the like.

[0055] As a method for detecting the gaze direction of the image capturer, known methods can be applied. For example, it is possible to apply a corneal reflection method, in which infrared light or the like is emitted from a light emitting unit and reflected from the cornea by a gaze direction detecting unit, which detects the gaze direction of the image capturer based on the position of the pupil. Furthermore, for example, a method can be applied that uses image recognition to identify a point that does not move, such as the inner or outer corner of the eye, and estimates the gaze direction based on the position of the iris of the eye.

[0056] The visual line detection unit 114 is provided, for example, in the EVF of the imaging device 100. The visual line detection unit 114 is provided, for example, in the housing of the imaging device 100. For example, the visual line detection unit 114 may be provided on the surface of the housing of the imaging device 100 on which the display unit 110 is provided.

[0057] The line-of-sight direction detected by the line-of-sight detection unit 114 is supplied to the control unit 101 .

[0058] The image capturer state determination unit 101A of the control unit 101 generates line of sight information based on the line of sight detection result supplied from the line of sight detection unit 114. The line of sight information is, for example, information indicating the distribution of the line of sight detection results. Specifically, Figure 2 As shown in , the line of sight information is the distribution of the line of sight detection results PA, which is the portion corresponding to the line of sight direction detected by the line of sight detection unit 114. The line of sight information is obtained for each appropriate area AR of the display unit 110. The area AR can be in units of pixels or in units of blocks of M pixels x N pixels (M and N are appropriate positive numbers). The image capturer state determination unit 101A determines the state of the image capturer based on the distribution of the line of sight detection results PA included in the line of sight information.

[0059] Based on the distribution of the gaze detection results PA, the image capturer state determination unit 101A determines whether the state of the image capturer is a state of gazing at a subject, that is, a state in which a subject serving as a gaze target has been determined (hereinafter referred to as a gaze target determined state as appropriate), or a state in which a gaze target has not yet been determined (hereinafter referred to as a gaze target undetermined state as appropriate). In this embodiment, the image capturer state determination unit 101A determines whether the state is a state in which a gaze target has not yet been selected (hereinafter referred to as a gaze target unselected state as appropriate), which is an example of a gaze target undetermined state, or a state in which the image capturer's degree of concentration in image capturing is lower than that in the gaze target undetermined state (hereinafter referred to as an idle state as appropriate).

[0060] Specific examples of the image capturer's state being a gaze target determination state include a state in which the image capturer is performing image capture while following a certain subject, or performing image capture in which a desired subject is in focus. Furthermore, a specific example of the image capturer's state being a gaze target unselected state includes a state in which it is difficult to determine which of a certain subject and another subject is in focus for image capture. Specific examples of the image capturer's state being an idle state include a state in which the image capturer is looking through the viewfinder while watching a bird's nest and waiting for the bird to return to the nest, as well as a state in which the image capturer is performing image capture during halftime of a football game or during a scene in which the offense and defense change during image capture of a baseball game.

[0061] The image capturer state determination unit 101A determines whether the state of the image capturer is any of the above three states based on the distribution of the line of sight detection results PA, for example. Figure 2 As shown in , the image capturer state determination unit 101A obtains a histogram of the values ​​of the distribution of the gaze detection results PA corresponding to both the X-axis direction and the Y-axis direction of the display unit 110, and obtains the variance corresponding to each peak thereof. Then, in the event that there is a peak whose variance is smaller than a predetermined value, the image capturer state determination unit 101A determines that the state of the image capturer is the gaze target determination state.

[0062] Furthermore, if the distribution of the gaze detection results does not have a peak with a variance value less than a predetermined value, the image capturer state determination unit 101A determines that the image capturer's state is the gaze target undetermined state. More specifically, if the distribution of the gaze detection results indicated by the gaze information only has a plurality of peaks with variance values ​​greater than a predetermined value, that is, if the variance values ​​of all peaks are greater than the predetermined value, the image capturer state determination unit 101A determines that the image capturer's state is the gaze target unselected state. If the distribution of the gaze detection results has a variance value greater than the predetermined value, the image capturer state determination unit 101A determines that the image capturer's state is the idle state.

[0063] Note that the image capturer state determination unit 101A can determine the state of the image capturer based on the trajectory of the user's line of sight detected by the line of sight detection unit 114. For example, in a case where the line of sight trajectory of the user falls within a certain range within a predetermined time, the image capturer state determination unit 101A determines that the state of the image capturer is the gaze target determined state. In addition, in a case where the line of sight trajectory of the user reciprocates between multiple positions on the display image within a predetermined time (for example, two or three or more positions can be used), the image capturer state determination unit 101A determines that the state of the image capturer is the gaze target undetermined state. In addition, in a case where the line of sight trajectory of the user is irregularly distributed within the predetermined time, the image capturer state determination unit 101A determines that the state of the user is an idle state. In other words, the idle state is the state of the image capturer corresponding to a case where the trajectory of the line of sight detection result indicated by the line of sight information is wider than the trajectory in the gaze target determined state.

[0064] Furthermore, the image capturer state determination unit 101A may determine the state of the image capturer based on the number of peaks having a sharpness equal to or greater than a predetermined value and appearing when the histogram of the distribution of the gaze detection results is smoothed. For example, the image capturer state determination unit 101A determines that the state of the image capturer is the gaze target determined state when the number of peaks having sharpness is 1, determines that the state of the image capturer is the gaze target unselected state when the number of peaks having sharpness is within the range of 2 to 4, and determines that the state of the image capturer is the idle state when the number of peaks having sharpness is 5 or more.

[0065] [Processing Flow]

[0066] (Overall processing flow)

[0067] Next, we will refer to Figure 3The flowchart describes the flow of processing (entire processing) performed by the imaging device 100. For example, when the power of the imaging device 100 is turned on or when the mode is moved to the imaging mode, the processing described below is performed. According to the detection result by the gyro sensor or the pressure-sensitive sensor, the processing can be started at a stage where the user is in a posture of lifting the imaging device 100 and capturing an image. In addition, the processing can be started after the line of sight detection unit 114 detects the line of sight. The processing can be repeated while the line of sight is detected. In addition, unless otherwise specified, the processing described below is performed by the control unit 101. The same applies similarly to the description of other processing flows.

[0068] In step ST11, the image capturer state determination unit 101A detects (generates) line of sight information based on the line of sight detection result of the line of sight detection unit 114. Then, the process proceeds to step ST12.

[0069] In step ST12, the image capturer state determination unit 101A determines the state of the image capturer based on the line of sight information. If the image capturer state is the gaze target determined state, the process proceeds to step ST13. If the image capturer state is the gaze target unselected state, the process proceeds to step ST14. If the image capturer state is the idle state, the process proceeds to step ST15.

[0070] In step ST13, the control unit 101 performs control in a manner corresponding to the state of the image capturer (i.e., the gaze target determination state). This control is control for performing assistance according to the state of the image capturer, and specific examples thereof include displaying an AF pointer and aligning the AF pointer with the gaze target subject or performing AF on the gaze target subject.

[0071] In step ST14, the control unit 101 performs control in a manner corresponding to the state of the image capturer (i.e., gaze target unselected state). In step ST15, the control unit 101 performs control in a manner corresponding to the state of the image capturer (i.e., gaze target idle state).

[0072] (Specific example of control corresponding to the gaze target unselected state)

[0073] Next, a specific example of control corresponding to the gaze target unselected state will be described. When the state of the image capturer is the gaze target unselected state, the control unit 101 determines the focus area so that the focus area includes the target subject. Note that including the target subject means including at least a portion of the target subject. In addition, determining the focus area means determining a partial area in the display unit 110 as the focus area. In addition, the subject is a subject identified by a known image recognition process. For example, the control unit 101 determines the target subject based on the imaging condition before the state of the image capturer becomes the gaze target unselected state. As an example, the control unit 101 determines the target subject based on the subject that was the gaze target before the state of the image captured image becomes the gaze target unselected state. Note that the subject occupying the largest area or the subject with the largest movement in the display image currently displayed on the display unit 110 can be determined as the target subject.

[0074] Specific examples will refer to Figure 4 Provide a description. Figure 4 A is a diagram illustrating an example of a captured image obtained in a state where the state of the image capturer is a gaze target determination state (an example of a display image displayed on the display unit 110). Figure 4 As shown in FIG. 1A , the image capturer is gazing at the batsman BH as the gaze target subject. The AF control unit 113 is controlled to bring the face of the batsman BH into focus. The AF frame 21 and the AF pointer 22 are displayed on the face of the batsman BH.

[0075] Here, it is assumed that the captured image is changed from Figure 4 The captured image shown in A is changed to Figure 4 The captured image shown in B. In this case, the display position of the batsman BH to be AF is changed as the viewing angle changes. Figure 4 In the captured image shown in B, the image capturer is in a state of being undecided about which subject to focus on, that is, the state of the image capturer is a gaze target unselected state. Generally, since it is assumed that the image capturer wants to continue focusing on the target that the image capturer has been gazing at so far (in this example, the face of the batsman BH), the control unit 101 controls the AF control unit 113 so that the focus is on the face of the batsman BH. According to this control, as Figure 4 As shown in C, an AF frame 21 and an AF pointer 22 are displayed on the face of the batsman BH.

[0076] Next, refer to Figure 5The flowchart below will describe an example of the control (processing) flow corresponding to the gaze target unselected state. For example, if the image capturer state determination unit 101A determines that the image capturer's state is the gaze target unselected state, the processing described below is executed. Furthermore, the processing described below can be repeated while the line of sight is detected.

[0077] In step ST21, the control unit 101 starts control in a manner corresponding to the case where the state of the image capturer is a gaze target unselected state. Then, the process proceeds to step ST22.

[0078] In step ST22, it is determined whether a subject in focus in the gaze target determination state (target subject in this example) is included in the currently captured image. If a subject in focus is included (yes), the process proceeds to step ST23.

[0079] In step ST23, control is performed to focus on the subject in focus in the gaze target determination state. The processing from step ST22 to step ST23 is the same as that in the above reference step. Figure 4 Processing described in A to 4C.

[0080] In the determination processing of step ST22 , in a case where the subject in focus in the gaze target determination state is not included in the currently captured image (No), the processing proceeds to step ST24 .

[0081] In step ST24 , it is determined whether the currently captured image includes the same type of subject as the subject in focus in the gaze target determination state by executing subject recognition processing, etc. If the same type of subject is included (Yes), the process proceeds to step ST25 .

[0082] In step ST25 , a subject of the same type as the subject in focus in the gaze target determination state is determined as a target subject, and control is performed to focus on the subject.

[0083] In the determination processing of step ST24 , in a case where the same type of subject as the target in focus in the gaze target determination state is not included in the currently captured image (No), the processing proceeds to step ST26 .

[0084] In step ST26 , the subject having the largest area among the subjects currently displayed on the display unit 110 is determined as the target subject, and control is performed to focus on the subject.

[0085] As a specific example of the processing related to step ST24, for example, when an image in which multiple people and multiple horses are mixed at a horse race is to be captured, and the state of the image capturer has changed to a gaze target unselected state, it is determined whether the type of object that was focused on before the state changed to a gaze target unselected state (for example, the type of animal including people, the type of attribute of people (such as gender and race), and the type of object (such as a table, a house, and a ball)) is included in the currently captured image. If a person is included, control is performed to focus on the person. If a person is not included, control is performed to focus on a subject having a large area (for example, a horse).

[0086] Note that, when the state of the image capturer is a gaze target unselected state, the control unit 101 may highlight the displayed target subject. Examples of highlight display include a display mode in which the brightness of the target subject is increased, the brightness around the target subject is decreased, the target subject blinks, and a mark is displayed on or near the target subject.

[0087] (Specific Example of Control Corresponding to Idle State)

[0088] Next, a specific example of control corresponding to the gaze target unselected state will be described. If the image capturer's state is determined to be idle, for example, control is executed to set the imaging device 100 to power saving mode by disabling some functions or reducing performance. For example, the display frame rate of the EVF display is reduced to switch the imaging device 100 to power saving mode. Furthermore, if the image capturer's state is idle, assume that the image capturer is searching for a scene with a subject while capturing an image of a sport in which many athletes participate. Therefore, to facilitate subject capture, control is executed to widen the distance measurement range and automatically widen the focus area (focus wide mode). Furthermore, if the image capturer's state changes from a state other than idle (e.g., a gaze target determined state or a gaze target unselected state) to idle, the angle of view can be made wider than in states other than idle. Furthermore, if the image capturer's state is idle, with a low level of concentration, there is a risk of missing important scenes while in idle mode. Therefore, control (image storage control) is executed to automatically store images continuously captured by image capture in a buffer memory. The control corresponding to the idle state will be described in more detail below.

[0089] (First Example)

[0090] Figure 6This is a flowchart illustrating a first example of the flow of control corresponding to the idle state. This first example is an example in which, when the image capturer's state is the idle state, the control unit 101 selects a control method corresponding to the imaging situation from among multiple control methods. More specifically, it is an example in which a control method corresponding to the imaging mode as the imaging situation is selected from among multiple control methods. Note that the following processing can be repeated while detecting a line of sight.

[0091] In step ST31, the image capturer state determination unit 101A determines that the state of the image capturer is the idle state, and the control unit 101 starts control in a manner corresponding to the idle state. Then, the process proceeds to step ST32.

[0092] In step ST32, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Note that the focus wide mode is not necessarily set (the same applies to other examples). Then, the process proceeds to step ST33.

[0093] In step ST33, the imaging mode of the imaging device 100 is determined. Here, if the imaging mode is the programmed automatic mode (hereinafter referred to as the P mode as appropriate), the process proceeds to step ST34. If the imaging mode is the shutter speed priority mode (hereinafter referred to as the S mode as appropriate), the process proceeds to step ST35. If the imaging mode is any other mode (for example, the automatic mode), the process proceeds to step ST36.

[0094] In step ST34, control is performed to set the wide angle of view mode for widening the angle of view, thereby performing image capture at a wider angle of view. Since the P mode is generally set when the image capturer is a beginner or a novice, the wide angle of view mode is set to assist such an image capturer.

[0095] In step ST35, control is executed to set a storage mode for storing images continuously captured by image capture in a buffer memory (e.g., a ring buffer). In the case of the S mode, since it is assumed that a scene with an image of a subject in relatively rapid motion is captured, the storage mode is set so that a decisive scene is not missed. The buffer length of the ring buffer can be increased according to the period of idle state.

[0096] In step ST36, control is performed to set a power saving mode for reducing power consumption in the imaging device 100. Specific examples of the power saving mode include control to limit the frequency band for output from the imaging element 104, control to reduce the brightness of the display unit 110, etc. in addition to the above-mentioned control.

[0097] (Second example)

[0098] Figure 7 This is a flowchart illustrating a second example of the flow of control corresponding to the idle state. This second example is an example in which, when the image capturer's state is the idle state, the control unit 101 selects a control method corresponding to the imaging situation from among multiple control methods. More specifically, it is an example in which a control method corresponding to the type of subject as the imaging situation is selected from among multiple control methods. Note that the following processing can be repeated while detecting a line of sight.

[0099] In step ST41, the image capturer state determination unit 101A determines that the state of the image capturer is the idle state, and the control unit 101 starts control in a manner corresponding to the idle state. Then, the process proceeds to step ST42.

[0100] In step ST42, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Then, the process proceeds to step ST43.

[0101] In step ST43, subject recognition processing is performed to determine the subject. If the subject includes a moving subject (e.g., a person, an animal, a vehicle, etc.), the process proceeds to step ST44. If the subject (e.g., only a landscape) does not include a moving subject, the process proceeds to step ST45.

[0102] In step ST44 , since the subject includes a moving subject, control is performed to set the storage mode so as not to miss the decisive scene. On the other hand, in step ST45 , since the subject does not include a moving subject, control is performed to set the power saving mode.

[0103] (Third Example)

[0104] Figure 8 This is a flowchart illustrating a third example of the flow of control corresponding to the idle state. This third example is an example in which, when the image capturer is in the idle state, the control unit 101 selects a control method corresponding to the duration of the idle state from among multiple control methods. Note that the following processing can be repeated while detecting line of sight.

[0105] In step ST51, the image capturer state determination unit 101A determines that the state of the image capturer is the idle state, and the control unit 101 starts control in a manner corresponding to the idle state. Then, the process proceeds to step ST52.

[0106] In step ST52, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Then, the process proceeds to step ST53.

[0107] In step ST53, it is determined whether the state in which the image capturer is in the idle state continues for a predetermined time (e.g., 1 minute to several minutes) or longer. If the state continues (yes), the process proceeds to step ST54. If the state does not continue (no), the process proceeds to step ST55.

[0108] In step ST54, control is performed to set the storage mode in the imaging device 100. In step ST55, control is performed to set the power saving mode in the imaging device 100.

[0109] As a specific case of this example, for example, it is assumed that an image of a bird or wild animal is being captured. In the case where the user is a professional photographer waiting for an animal, it is detected that the user is looking through the viewfinder and that the line of sight is present but not focused, that is, the state is an idle state. If a period of time (for example, 1 minute or longer) has passed in the idle state, control is executed to set the storage mode based on the assumption that the user is waiting for the animal but does not know when the animal will arrive at the location. On the other hand, if the idle state does not continue, control is executed to set the power saving mode based on the assumption that the user is not waiting.

[0110] (Fourth Example)

[0111] Figure 9 This is a flowchart illustrating a fourth example of the flow of control corresponding to the idle state. This fourth example is an example in which, when the image capturer's state is idle, the control unit 101 selects a control method corresponding to the imaging situation from among multiple control methods. More specifically, this is an example in which a control method corresponding to a change in the captured image as the imaging situation is selected from among multiple control methods. Note that the following processing can be repeated while detecting a line of sight.

[0112] In step ST61, the image capturer state determination unit 101A determines that the state of the image capturer is the idle state, and the control unit 101 starts control in a manner corresponding to the idle state. The process then proceeds to step ST62.

[0113] In step ST62, control is performed to widen the distance measurement range and automatically widen the focus area, and the setting of the imaging device 100 is set to the focus wide mode. Then, the process proceeds to step ST63.

[0114] In step ST63, it is determined whether the image change is within a threshold. For example, the image change is determined based on the correlation between the current frame and a past frame (e.g., a frame that is one frame earlier in time than the current frame). If the correlation is equal to or greater than a certain level, the image change is determined to be within the threshold. If the correlation is less than a certain level, the image change is determined to be greater than the threshold. If the image change is within the threshold (yes), the process proceeds to step ST64. Alternatively, if the image change is greater than the threshold, the process proceeds to step ST65.

[0115] In step ST64, control is performed to set the storage mode in the imaging device 100. In step ST65, control is performed to set the power saving mode in the imaging device 100.

[0116] As a specific case of this example, for example, it is assumed that an image of a bird or wild animal is captured, as in the third example described above. In the case where the person is a professional photographer waiting for an animal, it is detected that the user is looking at the viewfinder and the line of sight exists, but the line of sight is not focused, that is, the state is an idle state. In this case, generally speaking, most of the scenery being captured is captured, but the change in the image is small. Therefore, in the case where the change in the image is small, in order not to miss a decisive scene (for example, a scene in which an animal is captured), control is performed to set the storage mode. On the other hand, in the case where the change in the image is large, it is determined that the situation is not the above-mentioned case, and control is performed to set the power saving mode.

[0117] According to the present embodiment described above, it is possible to perform appropriate control not only in a state where the image capturer actively performs a predetermined operation but also in a state where the image capturer is in another state (specifically, the state of the image capturer is a gaze target unselected state or an idle state).

[0118] <Edit>

[0119] Although one embodiment of the present disclosure has been specifically described above, the contents of the present disclosure are not limited to the above embodiment, and various modifications based on the technical idea of ​​the present disclosure are possible.

[0120] In the above-described embodiment, there may be a case where control is not performed in a manner corresponding to the state of the image capturer determined based on the line of sight information. Figure 10 is a flowchart illustrating the flow of processing performed in this case. Note that the Figure 3 Description of content that overlaps with the content described in .

[0121] After the processing of step ST11, the processing related to step ST71 is performed. In step ST71, it is determined whether the image capturer is operating the settings of the imaging device 100. Examples of the operation of the settings include the operation of settings related to shutter speed and ISO sensitivity. Whether the settings of the imaging device 100 are being operated can be determined based on the presence or absence of an operation signal generated in response to the operation or the result of sensing by a sensor such as a dial provided in the input unit 109.

[0122] In the case of operating the setting in step ST71 (Yes), the process proceeds to step ST72, and a series of processes ends. In the case of not operating the setting in step ST71 (No), the process proceeds to step ST12. Since the process after step ST12 has been described, redundant description will be omitted.

[0123] When the image capturer performs an active setting operation, control is performed according to the image capturer's state, and the setting is performed according to the control, which may be different from the setting desired by the image capturer. In this example, since it is determined whether the setting-related operation is being performed, it is possible to avoid the above inconvenience.

[0124] Figure 11 is a flowchart illustrating the flow of processing performed in another case where control according to the state of the image capturer determined based on the line of sight information is not performed. After the processing of step ST11, processing related to step ST81 is performed.

[0125] In step ST81, it is determined whether the mode of the imaging device 100 is the manual mode (hereinafter referred to as the M mode as appropriate). If the mode of the imaging device 100 is the M mode (Yes), the process proceeds to step ST82, and the series of processes ends. If the mode of the imaging device 100 is not the M mode (No), the process proceeds to step ST12. Since the processes after step ST12 have already been described, redundant description will be omitted.

[0126] If the image capturer is a senior person, there may be cases where it is undesirable to perform control in a manner corresponding to the image capturer's status. In this case, the image capturer can avoid the above inconvenience by setting the mode of the imaging device 100 to M mode. Note that not only M mode but also a dedicated mode in which control according to the image capturer's status is not performed can be set.

[0127] In the above embodiment, a time factor may be considered when determining the state of the image capturer. For example, even if the state of the image capturer is determined to be an idle state based on line of sight information, control based on the idle state may be initiated if this state continues for a predetermined time (e.g., five seconds) or longer.

[0128] Note that, in the above-described embodiment, the image capturer state determination unit 101A may determine any one of the states of the image capturer.

[0129] In the processing according to the above-described embodiment, processing based on machine learning, that is, processing using a learning model obtained by learning performed in advance may be performed.

[0130] The configurations, methods, steps, shapes, materials, numerical values, etc. described in the above embodiments and modifications are merely examples, and configurations, methods, steps, shapes, materials, numerical values, etc. different from the above can be used as needed, or the configurations, methods, steps, shapes, materials, numerical values, etc. described in the above embodiments and modifications can be replaced with known ones. In addition, the configurations, methods, steps, shapes, materials, numerical values, etc. in the embodiments and modifications can be combined with each other within the scope that no technical contradiction occurs.

[0131] Note that the contents of the present disclosure should not be interpreted as being limited by the effects exemplified in this specification.

[0132] The present disclosure can also adopt the following configurations. (1)

[0134] An imaging device comprising

[0135] A control unit that performs control in a manner corresponding to the fact that the state of the image capturer determined based on the line of sight information is a gaze target undetermined state in which a gaze target has not yet been determined. (2)

[0137] (1) The imaging device described in

[0138] The gaze target undetermined state is a state of the image capturer corresponding to the fact that the distribution of the visual line detection results indicated by the visual line information does not have a peak with a variance value smaller than a predetermined value. (3)

[0140] The imaging device described in (1) or (2), wherein

[0141] The control unit determines a state of the image capturer based on the line of sight information. (4)

[0143] The imaging device according to any one of (1) to (3), wherein

[0144] In a case where the state of the image capturer is a gaze target unselected state in which a gaze target has not been selected as a gaze target undetermined state, the control unit determines the focus area so that the focus area includes the target subject. (5)

[0146] (4) The imaging device described in claim 4, wherein

[0147] In a case where the state of the image capturer is a gaze target unselected state, the control unit determines the target subject based on an imaging situation before the state of the image capturer becomes the gaze target unselected state. (6)

[0149] (5) The imaging device described in claim 5, wherein

[0150] The control unit determines the target subject based on a subject that is a gaze target before the state of the image capturer becomes a gaze target unselected state. (7)

[0152] The imaging device according to any one of (4) to (6), wherein

[0153] The gaze target unselected state is a state of the image capturer corresponding to the fact that the distribution of the visual line detection results indicated by the visual line information has only a plurality of peaks having a variance value greater than a predetermined value. (8)

[0155] The imaging device according to any one of (4) to (6), wherein

[0156] The gaze target unselected state is a state of the image capturer corresponding to the fact that the trajectory of the visual line detection result indicated by the visual line information reciprocates between a plurality of positions on the display image. (9)

[0158] The imaging device according to any one of (1) to (8), wherein

[0159] In a case where the state of the image capturer is an idle state as an undetermined state, when the state of the image capturer changes from a state other than the idle state to the idle state, the control unit makes the viewing angle wider than in a state other than the idle state. (10)

[0161] The imaging device according to any one of (1) to (8), wherein

[0162] In a case where the state of the image capturer is an idle state which is an undetermined state, the control unit performs power saving control. (11)

[0164] The imaging device according to any one of (1) to (8), wherein

[0165] In a case where the state of the image capturer is an idle state which is an undetermined state, the control unit performs image storage control. (12)

[0167] The imaging device according to any one of (1) to (8), wherein

[0168] In a case where the state of the image capturer is an idle state as an undetermined state, when the state of the image capturer changes from a state other than the idle state to the idle state, the control unit makes the distance measurement range wider than in a state other than the idle state. (13)

[0170] The imaging device according to any one of (1) to (8), wherein

[0171] In a case where the state of the image capturer is an idle state, the control unit selects a control method corresponding to the imaging situation from among a plurality of control methods. (14)

[0173] (13) The imaging device described in claim 13, wherein

[0174] The control unit selects a control method corresponding to the imaging mode as the imaging situation from among the plurality of control methods in a case where the state of the image capturer is an idle state. (15)

[0176] (13) The imaging device described in claim 13, wherein

[0177] In a case where the state of the image capturer is an idle state, the control unit selects a control method corresponding to the type of the subject as the imaging situation from among the plurality of control methods. (16)

[0179] (13) The imaging device described in claim 13, wherein

[0180] In a case where the state of the image capturer is an idle state, the control unit selects a control method corresponding to a duration of the idle state as the imaging situation from among the plurality of control methods. (17)

[0182] The imaging device according to any one of (9) to (16), wherein

[0183] The idle state is a state of the image capturer corresponding to the fact that the distribution of the visual line detection results indicated by the visual line information has a variance value larger than a predetermined value. (18)

[0185] The imaging device according to any one of (9) to (16), wherein

[0186] The idle state is a state of the image capturer corresponding to the fact that the locus of the visual line detection result indicated by the visual line information is wider than that in the gaze target determination state in which the gaze target has been determined. (19)

[0188] A method for controlling an imaging device, the method comprising:

[0189] The control unit is caused to perform control in a manner corresponding to the fact that the state of the image capturer determined based on the visual line information is a gaze target undetermined state in which a gaze target has not yet been determined. (20)

[0191] A program for causing a computer to execute a method of controlling an imaging device, the method comprising

[0192] The control unit is caused to perform control in a manner corresponding to the fact that the state of the image capturer determined based on the visual line information is a gaze target undetermined state in which a gaze target has not yet been determined.

[0193] Reference Signs List

[0194] 100 Imaging Device

[0195] 101 Control Unit

[0196] 101A Image capturer status determination unit

[0197] 114 Eyesight Detection Unit

Claims

1. An imaging device comprising: a control unit that performs control in a manner corresponding to the fact that the state of the image capturer determined based on the line of sight information is a gaze target undetermined state in which a gaze target has not yet been determined, The control unit is configured to determine whether the gaze target undetermined state is a gaze target unselected state in which a gaze target has not been selected or an idle state. The sight line information includes the distribution of sight line detection results. wherein the gaze target unselected state corresponds to a distribution of gaze detection results including only a plurality of peaks having corresponding variance values ​​greater than a predetermined value, and The idle state corresponds to an irregular distribution of sight line detection results including a variance value greater than the predetermined value.

2. The imaging device according to claim 1, wherein The gaze target undetermined state is a state of the image capturer corresponding to the fact that the distribution of the visual line detection results indicated by the visual line information does not have a peak with a variance value smaller than a predetermined value.

3. The imaging device according to claim 1, wherein The control unit determines a state of the image capturer based on the line of sight information.

4. The imaging device according to claim 1, wherein In a case where the state of the image capturer is a gaze target unselected state in which a gaze target has not been selected as the gaze target undetermined state, the control unit determines a focus area such that the focus area includes a target subject.

5. The imaging device according to claim 4, wherein In a case where the state of the image capturer is the gaze target unselected state, the control unit determines the target subject based on an imaging situation before the state of the image capturer becomes the gaze target unselected state. The imaging device according to claim 5 , wherein The control unit determines the target subject based on a subject that is the gaze target before the state of the image capturer changes to the gaze target unselected state.

7. The imaging device according to claim 4, wherein The gaze target unselected state is a state of the image capturer corresponding to the fact that the distribution of the visual line detection results indicated by the visual line information has only a plurality of peaks having a variance value greater than a predetermined value.

8. The imaging device according to claim 4, wherein The gaze target unselected state is a state of the image capturer corresponding to the fact that the trajectory of the visual line detection result indicated by the visual line information reciprocates between a plurality of positions on a display image.

9. The imaging device according to claim 1, wherein In a case where the state of the image capturer is an idle state as the undetermined state, when the state of the image capturer changes from a state other than the idle state to the idle state, the control unit makes the angle of view wider than in a state other than the idle state.

10. The imaging device according to claim 1, wherein The control unit performs power saving control in a case where the state of the image capturer is an idle state as the undetermined state.

11. The imaging device according to claim 1, wherein The control unit performs image storage control in a case where the state of the image capturer is an idle state as the undetermined state.

12. The imaging device according to claim 1, wherein In a case where the state of the image capturer is an idle state as the undetermined state, when the state of the image capturer changes from a state other than the idle state to the idle state, the control unit makes the distance measurement range wider than in a state other than the idle state.

13. The imaging device according to claim 1, wherein The control unit selects a control method corresponding to an imaging situation from among a plurality of control methods in a case where the state of the image capturer is an idle state.

14. The imaging device according to claim 13, wherein In a case where the state of the image capturer is the idle state, the control unit selects a control method corresponding to an imaging mode as the imaging situation from among the plurality of control methods.

15. The imaging device according to claim 13, wherein In a case where the state of the image capturer is the idle state, the control unit selects a control method corresponding to a type of the subject as the imaging situation from among the plurality of control methods.

16. The imaging device according to claim 13, wherein In a case where the state of the image capturer is the idle state, the control unit selects a control method corresponding to a duration of the idle state as the imaging situation from among the plurality of control methods.

17. The imaging device according to claim 9, wherein The idle state is a state of the image capturer corresponding to the fact that the distribution of the visual line detection results indicated by the visual line information has a variance value larger than a predetermined value.

18. The imaging device according to claim 9, wherein The idle state is a state of the image capturer corresponding to the fact that a locus of a visual line detection result indicated by the visual line information is wider than in a gaze target determination state in which a gaze target has been determined.

19. A method for controlling an imaging device, the method comprising: causing the control unit to perform control in a manner corresponding to the fact that the state of the image capturer determined based on the line of sight information is a gaze target undetermined state in which a gaze target has not yet been determined, The control unit is configured to determine whether the gaze target undetermined state is a gaze target unselected state in which a gaze target has not been selected or an idle state. The sight line information includes the distribution of sight line detection results. wherein the gaze target unselected state corresponds to a distribution of gaze detection results including only a plurality of peaks having corresponding variance values ​​greater than a predetermined value, and The idle state corresponds to an irregular distribution of sight line detection results including a variance value greater than the predetermined value.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, a method for controlling an imaging device is implemented, the method comprising: causing the control unit to perform control in a manner corresponding to the fact that the state of the image capturer determined based on the line of sight information is a gaze target undetermined state in which a gaze target has not yet been determined, The control unit is configured to determine whether the gaze target undetermined state is a gaze target unselected state in which a gaze target has not been selected or an idle state. The sight line information includes the distribution of sight line detection results. wherein the gaze target unselected state corresponds to a distribution of gaze detection results including only a plurality of peaks having corresponding variance values ​​greater than a predetermined value, and The idle state corresponds to an irregular distribution of sight line detection results including a variance value greater than the predetermined value.

Images