Imaging device

Abstract

To provide a technique capable of reducing power consumption of an imaging apparatus capable of detecting a user's line of sight.SOLUTION: An imaging apparatus has imaging means, line-of-sight detection means for detecting a user's line of sight, subject detection means for detecting a subject from a captured image by the imaging means, and control means for changing the state of the line-of-sight detection means to a second state in which the power consumption of the line-of-sight detection means is lower than in the first state when a subject of a specific type is detected by the subject detection means when the state of the line-of-sight detection means is in the first state in which the line-of-sight detection means detects the line-of-sight.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an imaging device, and more particularly to a technique for detecting the line of sight of a user of the imaging device.

Background Art

[0002] As functions of imaging devices such as digital cameras, a main subject determination function, a subject tracking function, a line-of-sight detection function, etc. are known. The main subject determination function is a function of selecting (determining) a main subject to be targeted for autofocus (AF) or tracking from one or more subjects included in the captured image. The subject tracking function is a function of tracking a subject, and the line-of-sight detection function is a function of detecting the line of sight of a user of the imaging device.

[0003] Patent Document 1 discloses a technique for determining a subject that has performed a specific operation as the main subject. Patent Document 2 discloses a technique for detecting the position toward which the line of sight of a user of an imaging device is directed, detecting the features of the subject existing at the detected position, and tracking the subject based on the detected features. Patent Document 3 discloses a technique for irradiating infrared light onto the eyeball, imaging the reflected light from the eyeball, detecting the position of the Purkinje image and the center position of the pupil from the captured image, and detecting the direction of the line of sight based on the detected positions.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to detect the movement of the line of sight with high precision, it is necessary to shorten the execution period of the line-of-sight detection. However, if the execution period of the line-of-sight detection is shortened, the power consumption of the imaging device will increase.

[0006] An object of the present invention is to provide a technique capable of reducing the power consumption of an imaging device capable of detecting a user's line of sight.

Means for Solving the Problems

[0007] A first aspect of the present invention includes an imaging unit, a line-of-sight detection unit that detects a user's line of sight, a subject detection unit that detects a subject from an image captured by the imaging unit, and when the state of the line-of-sight detection unit is a first state for detecting the line of sight, if a specific type of subject is detected by the subject detection unit, a control unit that changes the state of the line-of-sight detection unit to a second state in which the power consumption of the line-of-sight detection unit is smaller than that in the first state. An imaging device characterized by having

[0008] A second aspect of the present invention includes an imaging unit, a line-of-sight detection unit that detects a user's line of sight, a subject detection unit that detects a subject from an image captured by the imaging unit, and when the state of the line-of-sight detection unit is a first state for detecting the line of sight, even if a plurality of human bodies and one or more balls are detected by the subject detection unit, the state of the line-of-sight detection unit is not changed, and when one human body and one or more balls are detected by the subject detection unit, a control unit that changes the state of the line-of-sight detection unit to a second state in which the power consumption of the line-of-sight detection unit is smaller than that in the first state. An imaging device characterized by having

[0009] A third aspect of the present invention is a control method for an imaging device having imaging means and line-of-sight detection means for detecting a user's line of sight, the method comprising: detecting a subject from an image captured by the imaging means; and when the state of the line-of-sight detection means is a first state for detecting the line of sight, if a specific type of subject is detected from the image, changing the state of the line-of-sight detection means to a second state in which the power consumption of the line-of-sight detection means is lower than that in the first state.

[0010] A fourth aspect of the present invention is a control method for an imaging device having imaging means and line-of-sight detection means for detecting a user's line of sight, the method comprising: detecting a subject from an image captured by the imaging means; and when the state of the line-of-sight detection means is a first state for detecting the line of sight, even if a plurality of human bodies and one or more balls are detected from the image, not changing the state of the line-of-sight detection means, and when one human body and one or more balls are detected from the image, changing the state of the line-of-sight detection means to a second state in which the power consumption of the line-of-sight detection means is lower than that in the first state.

[0011] A fifth aspect of the present invention is a program for causing a computer to function as each means of the imaging device described above. A sixth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each means of the imaging device described above.

Advantages of the Invention

[0012] According to the present invention, it is possible to reduce the power consumption of an imaging device capable of detecting a user's line of sight.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0014] <Embodiment 1> Hereinafter, Embodiment 1 of the present invention will be described with reference to the drawings. The basic operation of the imaging device according to Embodiment 1 of the present invention will be described. The imaging device according to Embodiment 1 detects that the user of the imaging device has looked into the finder (viewfinder) (the user has put their eye to the finder). Then, in the eye-attached state (the state where the user has their eye to the finder), the imaging device uses the line-of-sight detection unit to detect the user's line of sight. Based on the detected line of sight, the imaging device selects a subject to be tracked from the captured image (the image of the current frame), and when displaying the images from the next frame onwards, generates an image of a tracking frame that surrounds the subject to be tracked, and superimposes and displays it on the captured image.

[0015] FIG. 1 is a block diagram schematically showing an example of the internal structure of the imaging device 100 according to Embodiment 1.

[0016] The first optical system 101 has a plurality of lenses including a movable lens such as a focus lens and forms an optical image of the imaging range on the imaging surface of the first imaging element 102.

[0017] The central control unit 122 is, for example, a CPU (Central Processing Unit), which expands and executes a program stored in the ROM (Read Only Memory) 124 in the RAM (Random Access Memory) 123. The central control unit 122 realizes the functions of the imaging device 100 by controlling the operations of the respective components of the imaging device 100. The ROM 124 is, for example, a rewritable non-volatile memory, which stores programs executable by the central control unit 122, set values, data such as a GUI (Graphical User Interface), and the like. The RAM 123 is, for example, a system memory used to expand a program executed by the central control unit 122 and store values necessary during the execution of the program. Although omitted in FIG. 1, the central control unit 122 is communicably connected to the respective components of the imaging device 100.

[0018] The first imaging element 102 is, for example, a CMOS image sensor having a color filter in a primary color Bayer array. In the first imaging element 102, a plurality of pixels each having a photoelectric conversion region are two-dimensionally arranged. The first imaging element 102 converts the optical image formed by the first optical system 101 into an electric signal group (analog image signal) by the plurality of pixels. The analog image signal is converted into a digital image signal (image data) by an A / D converter included in the first imaging element 102 and output. The A / D converter may be provided outside the first imaging element 102.

[0019] The evaluation value generation unit 114 generates signals and evaluation values for use in autofocus detection (AF), evaluation values for use in automatic exposure control (AE), etc. from the image data obtained from the first imaging device 102. The evaluation value generation unit 114 outputs the generated signals and evaluation values to the central control unit 122. The central control unit 122 controls the position of the focus lens of the first optical system 101 based on the signals and evaluation values obtained from the evaluation value generation unit 114, the fixation point position obtained from the gaze detection memory 125, and the tracking result obtained from the tracking unit 116. Also, the central control unit 122 determines shooting conditions (exposure time, aperture value, ISO sensitivity, etc.) based on this information. The evaluation value generation unit 114 may generate signals and evaluation values from the display image data generated by the post-processing unit 106 described later.

[0020] The first preprocessing unit 103 applies color interpolation processing to the image data obtained from the first imaging device 102. The color interpolation processing is a process of converting each of a plurality of pixel data (pixel values) constituting the image data into pixel data having values of an R component, a G component, and a B component, and is also called demosaicing processing. The first preprocessing unit 103 may perform reduction processing for reducing the number of pixels as necessary in order to reduce the processing load. The first preprocessing unit 103 stores the image data to which the processing has been applied in the display memory 104.

[0021] The first image correction unit 105 applies correction processing such as white balance correction processing and shading correction processing, and conversion processing from RGB format to YUV format, etc. to the image data stored in the display memory 104. The first image correction unit 105 outputs the image data to which the processing has been applied to the post-processing unit 106. The first image correction unit 105 may use the image data of one or more frames different from the frame to be processed among the image data stored in the display memory 104 for the correction processing. For example, the first image correction unit 105 may use the image data of frames before and / or after the frame to be processed in time series for the correction processing.

[0022] The post - processing unit 106 generates recording image data and display image data from the image data obtained from the first image correction unit 105. For example, the post - processing unit 106 applies an encoding process to the image data and generates a data file storing the encoded image data as the recording image data. The post - processing unit 106 outputs the recording image data to the recording unit 107. The display image data is the image data for display on the display unit 118 and has a size corresponding to the display size on the display unit 118. The post - processing unit 106 outputs the display image data to the information overlay unit 117.

[0023] The recording unit 107 records the recording image data obtained from the post - processing unit 106 on the recording medium 108. The recording medium 108 is, for example, a semiconductor memory card or a built - in non - volatile memory.

[0024] The second pre - processing unit 109 applies color interpolation processing (demosaicing processing) to the image data obtained from the first imaging device 102. The second pre - processing unit 109 may perform a reduction process to reduce the number of pixels as necessary, for example, to reduce the processing load. The second pre - processing unit 109 stores the image data to which the processing has been applied in the tracking memory 110. The tracking memory 110 and the display memory 104 may be different address spaces within the same memory. The first pre - processing unit 103 and the second pre - processing unit 109 may be the same (common) pre - processing unit.

[0025] The second image correction unit 130 applies correction processes such as white balance correction processing and shading correction processing, and conversion processing from RGB format to YUV format, etc. to the image data stored in the tracking memory 110. The second image correction unit 130 stores the image data to which the processes have been applied in the tracking memory 110. The second image correction unit 130 may perform image processing for obtaining image data suitable for subject detection processing. For example, when the representative luminance of the image data (e.g., the average luminance of all pixels) is equal to or less than a predetermined threshold value, the second image correction unit 130 may multiply all pixel data of the image data by the same coefficient (gain) so that image data having a representative luminance higher than the predetermined threshold value can be obtained. The second image correction unit 130 may use the image data of one or more frames different from the frame to be processed among the image data stored in the tracking memory 110 for the correction process. For example, the second image correction unit 130 may use the image data of frames before and / or after the frame to be processed in time series for the correction process.

[0026] Note that components related to the subject tracking function, such as the second preprocessing unit 109 and the second image correction unit 130, do not have to operate when the subject tracking function is not implemented. The image data to which the subject tracking function is applied is moving image data captured for live view display or recording, and is, for example, moving image data having a predetermined frame rate such as 30 fps, 60 fps, or 120 fps.

[0027] The first detection unit 112 detects one or more regions of a predetermined type of subject (subject regions) from the image data for one frame stored in the tracking memory 110 by the second image correction unit 130. The first detection unit 112 stores the detection result of the subject region (subject detection result) in the tracking memory 110. For example, the subject detection result includes the following information for each subject region. Also, the subject detection result includes the number of detected subject regions for each type of subject (object class described later). · The position and size of the subject region · An object class indicating the type of subject (such as car, airplane, train, bird, insect, human body, head, pupil, face, cat, dog, flower, fish, ball, etc.) ·Object state information 1 indicating whether or not a pupil is detected within the subject area ·Object state information 2 indicating whether or not a face and / or a head is detected within the subject area

[0028] The object state information 2 may be information indicating a face orientation such as "forward", "backward", "sideways", etc. The subject detection result may include the reliability of the information included in the subject detection result. Since the head, face, pupil, etc. are parts of a human body, a cat, a dog, etc., they can also be referred to as partial subjects And the object classes such as "head", "face", "pupil", etc. can also be referred to as partial object classes. The first detection unit 112 may detect a wearable such as a helmet as a subject on which the wearable is worn. For example, the first detection unit 112 may detect a helmet as a head

[0029] The first detection unit 112 can detect the subject area by using a known technique for detecting a feature area such as a face area of a person or an animal. For example, a learned class discriminator using learning data may be used as the first detection unit 112. There is no particular limitation on the algorithm for identification (classification). The first detection unit 112 may be realized by training a discriminator implementing multi-class logistic regression, support vector machine, random forest, neural network, etc

[0030] The second optical system 119 has an eyepiece lens and an optical path splitting prism. The second optical system 119 includes the eyeball of the user (the user of the imaging device 100) looking through the eyepiece lens within the imaging range, separates the infrared light in the imaging range with the optical path splitting prism, and forms an optical image composed of the separated infrared light on the imaging surface of the second imaging element 120. Thereby, an image including the user's eyeball can be captured. Further, the second optical system 119 projects the image displayed on the display unit 118 onto the retina of the user looking through the eyepiece lens. Thereby, the second optical system 119 also functions as a viewfinder, and the user can visually recognize the image displayed on the display unit 118

[0031] The eye contact detection unit 128 is arranged near the finder (the second optical system 119) together with the third optical system 127, regards an object close to the finder as the user's eyeball, and detects eye contact with the finder. The eye contact detection unit 128 includes a second infrared light irradiation unit 129 and an infrared light sensing unit 131.

[0032] The second infrared light irradiation unit 129 is an irradiation unit for irradiating the user (the eyeball in contact with the finder; the eyeball looking into the finder) with infrared light, and is, for example, an infrared light emitting diode. When there is an obstacle such as an eyeball near the finder, the infrared light emitted from the second infrared light irradiation unit 129 is irradiated onto the obstacle through the third optical system 127, reflected by the obstacle, and enters the infrared light sensing unit 131 through the third optical system 127.

[0033] The infrared light sensing unit 131 senses the incident infrared light, and calculates the distance between the eye contact detection unit 128 and the obstacle from the time difference from the timing when the second infrared light irradiation unit 129 irradiates the infrared light to the timing when the infrared light sensing unit 131 senses the infrared light. The eye contact detection unit 128 determines whether it is in an eye contact state (a state where the user is in contact with the finder) according to the calculated distance. The infrared light sensing unit 131 notifies the line-of-sight detection unit control unit 115 of information (the determination result) as to whether it is in an eye contact state.

[0034] The line-of-sight detection unit control unit 115 controls the operation of the line-of-sight detection unit that detects the user's line of sight. For example, when receiving a notification as to whether it is in an eye contact state from the eye contact detection unit 128, if it is in an eye contact state, the line-of-sight detection unit control unit 115 enables the operation of the line-of-sight detection unit so that line-of-sight detection (an operation of detecting the user's line of sight) is performed. If it is not in an eye contact state, the line-of-sight detection unit control unit 115 disables (stops) the operation of the line-of-sight detection unit so that line-of-sight detection is not performed. The line-of-sight detection unit includes, for example, a first infrared light irradiation unit 111, a second imaging element 120, a third preprocessing unit 121, and a second detection unit 126.

[0035] The first infrared light irradiation unit 111 is an irradiation unit for irradiating infrared light onto the user (the eyeball looking into the viewfinder; the eyeball peeking into the viewfinder), and is, for example, an infrared light emitting diode. When the user is looking into the eyepiece lens of the second optical system 119, the infrared light emitted from the first infrared light irradiation unit 111 is reflected by the eyeball of the user looking into the eyepiece lens. Then the infrared light reflected by the eyeball is separated by the beam splitting prism of the second optical system 119 and forms an image on the second imaging element 120.

[0036] The second imaging element 120 is, for example, a CMOS image sensor having a color filter that transmits infrared light. In the second imaging element 120, a plurality of pixels each having a photoelectric conversion region are two-dimensionally arranged. The second imaging element 120 converts the optical image formed by the second optical system 119 separating the infrared light with the beam splitting prism into an electrical signal group (analog image signal) by a plurality of pixels. The analog image signal is converted into a digital image signal (image data) by an A / D converter included in the second imaging element 120 and output. The A / D converter may be provided outside the second imaging element 120. When the user is looking into the eyepiece lens of the second optical system 119, image data of the user (specifically, the eyeball of the user looking into the eyepiece lens) is obtained by the second imaging element 120.

[0037] The third preprocessing unit 121 applies filter processing for removing signal components having a specific spatial frequency band, reduction processing for reducing the number of pixels, etc. to the image data obtained from the second imaging element 120. The third preprocessing unit 121 stores the image data to which the processing has been applied in the gaze detection memory 125.

[0038] The second detection unit 126 detects the user's line of sight based on the image captured by the second imaging device 120. Various conventional techniques can be used for the method of detecting the line of sight. For example, the second detection unit 126 reads out the image data stored in the line-of-sight detection memory 125, and detects the Purkinje image and the iris from the read image data. The Purkinje image is an image of the infrared light emitted from the first infrared light irradiation unit 111 and reflected by the cornea, and is also called a corneal reflection image. The first infrared light irradiation unit 111 has a plurality of infrared light sources (for example, four infrared light-emitting diodes), and the second detection unit 126 detects a plurality of Purkinje images corresponding to the plurality of infrared light sources respectively. Then, the second detection unit 126 detects the user's line of sight based on the position of the Purkinje image and the center position of the iris, and stores the detection result of the line of sight in the line-of-sight detection memory 125. For example, the second detection unit 126 calculates the position on the display unit 118 that the user is gazing at based on the position of the Purkinje image and the center position of the iris. Then, the second detection unit 126 converts the calculated position into the position on the image data, and stores the converted position in the line-of-sight detection memory 125 as the fixation point position which is the detection result of the line of sight. Note that other information such as the position before conversion and the angle indicating the direction of the line of sight may be stored as the detection result of the line of sight.

[0039] The target determination unit 113 determines any one of the one or more subject regions detected by the first detection unit 112 as the tracking target. For example, the target determination unit 113 reads out the fixation point position stored in the line-of-sight detection memory 125, and determines the subject region closest to the fixation point position as the tracking target. The target determination unit 113 may determine whether the object class of the subject region is a specific object class. Then, the target determination unit 113 may determine a subject region of a specific object class (a subject of a specific type) as the tracking target without using the fixation point position. The specific object class (specific type) will be described later. When there are a plurality of subject regions of a specific object class, the target determination unit 113 may determine the subject region closest to the fixation point position among the plurality of subject regions as the tracking target. The target determination unit 113 stores the information indicating the subject region determined as the tracking target in the tracking memory 110 as the tracking target information.

[0040] The trailing unit 116 estimates, as a trailing region, a region corresponding to the subject region of the trailing target from the image data of the frame to be processed (current frame) based on the trailing target information stored in the trailing memory 110 (trailing process of trailing the subject (region)). The method for estimating the trailing region is not particularly limited. For example, using the image data of the current frame and the image data of a past frame (e.g., the frame immediately before the current frame) captured before the current frame, the region of the current frame corresponding to the subject region of the trailing target is estimated as the trailing region. Then, the trailing unit 116 obtains the position and size of the estimated trailing region as the trailing result, and outputs the trailing result to the central control unit 122 and the information superposition unit 117. Here, the subject region of the current frame determined by the target determination unit 113 as the trailing target is not the region of the current frame estimated by the trailing unit 116. The trailing unit 116 estimates the region of the current frame corresponding to the subject region of the past frame determined by the target determination unit 113 as the trailing target. The trailing target information obtained when the target determination unit 113 determines the subject region of the current frame as the trailing target is used when the trailing unit 116 performs the trailing process for the next frame.

[0041]

[0042] Note that the target determination unit 113 may update the trailing target information for each frame, or may not update the trailing target information until determining the region of another subject as the trailing target, and may update the trailing target information when determining the region of another subject as the trailing target. When the trailing unit 116 performs the trailing process based on the similarity to the color composition information of the subject region of the trailing target, the trailing unit 116 may update the color composition information of the subject region of the trailing target with the color composition information of the estimated trailing region and use it for the trailing process of the next frame. The trailing unit 116 may estimate the trailing region by pattern matching using the subject region of the trailing target as a template. The trailing unit 116 may estimate the position and size of the trailing region using a multi-layer neural network including a learned convolutional layer.

[0043] ​The information superimposing unit 117 generates an image of the tracking frame based on the size of the tracking area estimated by the trailing part 116. For example, the image of the tracking frame is an image of a rectangular frame circumscribing the tracking area. Then, the information superimposing unit 117 applies a compositing process of superimposing the image of the tracking frame on the display image data obtained from the post-processing unit 106 so that the tracking frame is displayed at the position of the tracking area, based on the position of the tracking area, to generate composite image data. The information superimposing unit 117 may generate an image representing the current setting values and state of the imaging device 100, etc., and superimpose the generated image by the compositing process so that the generated image is displayed at a predetermined position. The information superimposing unit 117 outputs the composite image data to the display unit 118.

[0044] The display unit 118 is, for example, a liquid crystal display or an organic EL display, and displays an image based on the composite image data output by the information superimposing unit 117.

[0045] Through the series of processes described above, live view display for one frame is performed.

[0046] Here, the gaze detection unit control unit 115 reads out the subject detection result stored in the tracking memory 110 by the first detection unit 112 in the eye-attached state, and performs further control of the gaze detection unit based on the read subject detection result.

[0047] FIG. 2 shows an example of an image captured by the first imaging element 102 and displayed on the display unit 118, the fixation point position, the tracking frame, the AF point, etc. The AF point is a position where autofocus is performed.

[0048] The image 200 is an image captured by the first imaging device 102 and displayed on the display unit 118. The human body 201 and the face 202 of the human body 201 are shown in the image 200. The left pupil 203 and the right pupil 204 in the face 202 are also shown in the image 200. The first detection unit 112 detects, from the image 200 stored in the tracking memory 110, as a subject region, a region circumscribing the human body 201. Then, the first detection unit 112 generates, as information regarding the subject region of the human body 201, the position and size of the subject region of the human body 201, the object class "human body", object state information 1 "with pupils", object state information 2 "with face and head", etc. Further, the first detection unit 112 detects, from the image 200, as a subject region, a region circumscribing the face 202. Then, the first detection unit 112 generates, as information regarding the subject region of the face 202, the position and size of the subject region of the face 202, the object class "face", object state information 1 "with pupils", object state information 2 "with face and head", etc. Also, the first detection unit 112 detects, from the image 200, as a subject region, a region circumscribing the left pupil 203, and generates, as information regarding the subject region of the pupil 203, the position and size of the subject region of the pupil 203, the object class "pupil", etc. Also, the first detection unit 112 detects, from the image 200, as a subject region, a region circumscribing the right pupil 204, and generates, as information regarding the subject region of the pupil 204, the position and size of the subject region of the pupil 204, the object class "pupil", etc. The first detection unit 112 stores the subject detection result including the information regarding these subject regions in the tracking memory 110. The target determination unit 113 determines the tracking target based on the subject detection result stored in the tracking memory 110 and the gaze point position stored by the second detection unit 126 in the gaze detection memory 125. Here, it is assumed that the left pupil, the right pupil, the face, and the human body are predetermined in descending order of priority for determining the tracking target. Therefore, the target determination unit 113 determines the left pupil 203 (specifically, the subject region of the pupil 203) as the tracking target for the subsequent frames. Regarding the information, the position and size of the subject region of the face 202, the object class "face", object state information 1 "with pupils", object state information 2 "with face and head", etc. are generated. Also, the first detection unit 112 detects, from the image 200, as a subject region, a region circumscribing the left pupil 203, and generates, as information regarding the subject region of the pupil 203, the position and size of the subject region of the pupil 203, the object class "pupil", etc. Also, the first detection unit 112 detects, from the image 200, as a subject region, a region circumscribing the right pupil 204, and generates, as information regarding the subject region of the pupil 204, the position and size of the subject region of the pupil 204, the object class "pupil", etc. The first detection unit 112 stores the subject detection result including the information regarding these subject regions in the tracking memory 110. The target determination unit 113 determines the tracking target based on the subject detection result stored in the tracking memory 110 and the gaze point position stored by the second detection unit 126 in the gaze detection memory 125. Here, it is assumed that the left pupil, the right pupil, the face, and the human body are predetermined in descending order of priority for determining the tracking target. Therefore, the target determination unit 113 determines the left pupil 203 (specifically, the subject region of the pupil 203) as the tracking target for the subsequent frames.

[0049] Image 210 is an image several frames after image 200. Since it is several frames after, the position and size of the human body 211 in image 210 are different from those of the human body 201 in image 200. The tracking unit 116 performs pattern matching using the area of the left pupil 203 determined by the target determination unit 113 as a template in the past frame (the frame of image 200), and estimates the area of the left pupil (tracking area) in image 210. The information superposition unit 117 generates a tracking frame 215 based on the tracking result (the position and size of the tracking area). Then, the central control unit 122 determines the AF point based on the tracking result output by the tracking unit 116 and the fixation point position stored by the second detection unit 126 in the gaze detection memory 125. Here, it is assumed that the left pupil position (tracking result), right pupil position (tracking result), face position (tracking result), fixation point position, and human body position (tracking result) are predetermined in descending order of priority for determining the AF point. Therefore, the central control unit 122 determines the center position of the tracking area estimated as the area of the left pupil (the center position of the tracking frame 215) as the AF point, and executes focus lens control based on the distance calculation result to the target object at the AF point. The priority for determining the AF point is not particularly limited. For example, the priority of the face position may be lower than the priority of the fixation point position. In that case, if the pupil is not tracked, the fixation point position 216 is determined as the AF point.

[0050] Image 220 is an image several frames after image 210. Since it is several frames later, in image 220, the human body 221 is in a state where the face is blocked by the tree 225. Assume that in the frame immediately before image 220, the tracking unit 116 failed to track the pupil 203 (pattern matching using the area of the pupil 203 as a template), and the target determination unit 113 reset the human body as the tracking target. Therefore, in the frame of image 220, the area (tracking area) estimated by the tracking unit 116 is the area of the human body 221. The information superposition unit 117 generates a tracking frame 222 based on the tracking result (the position and size of the tracking area). Then, the central control unit 122 determines the AF point. Here, assume that the priorities for determining the AF point are, in descending order, the position of the left pupil (tracking result), the position of the right pupil (tracking result), the position of the face (tracking result), the fixation point position, and the position of the human body (tracking result). Therefore, the central control unit 122 determines the same position 223 as the fixation point position 224 as the AF point, and executes focus lens control based on the distance calculation result to the target object at the AF point.

[0051] Here, the gaze detection unit control unit 115 controls the operation of the gaze detection unit to be enabled or disabled based on the subject detection result obtained by the first detection unit 112. For example, when a position (tracking result) with a higher priority for determining the AF point than the fixation point position is obtained, the fixation point position is not determined as the AF point. Therefore, the gaze detection unit control unit 115 disables the operation of the gaze detection unit so as to reduce the power consumption of the imaging device 100. effectively.

[0052] Figure 3 is a flowchart showing an example of the operation of the imaging device 100. Figure 3 shows an example of the operation of controlling the gaze detection unit based on the subject detection result.

[0053] In step S301, the gaze detection unit control unit 115 determines whether a subject of a specific type is detected based on the subject detection result, specifically, whether a subject area of a specific object class is detected. If the gaze detection unit control unit 115 detects a subject area of a specific object class, the process proceeds to step S302. If no subject area of a specific object class is detected, the process proceeds to step S312. The specific object class is an object class of a predetermined tracking candidate (candidate for a tracking target), for example, an object class indicating a subject that may move. Here, it is assumed that partial object classes such as "head", "face", and "pupil" are not included in the specific object class. Subjects that may move include, for example, organisms belonging to mammals (human body, cat, dog, horse, etc.), organisms belonging to birds, organisms belonging to reptiles, organisms belonging to fish, organisms belonging to crustaceans, insects, vehicles (railway vehicles, automobiles (four-wheel automobiles, two-wheel automobiles, etc.), airplanes, ships, etc.), and the like. The specific object class is, for example, "human body", "cat", "dog", "bird", "horse", "other animals", "reptiles", "fish", "crustaceans", "insects", "train", "automobile", "motorcycle", "airplane", "helicopter", "ship", "other vehicles", and the like.

[0054] Note that when a subject area of a specific object class is detected, the gaze detection unit control unit 115 may also proceed to step S306. In that case, the subject area of the specific object class may be determined as the tracking target. When a plurality of subject areas of a specific object class are detected, any one of the plurality of subject areas may be selected and determined as the tracking target. The method for selecting the subject area is not particularly limited. For example, any one of the plurality of subject areas may be randomly selected, or any one of the plurality of subject areas may be selected based on parameters such as the size and position of each subject area. When no subject area of a specific object class is detected, the gaze detection unit control unit 115 may also proceed to step S307.

[0055] In step S302, the gaze detection unit control unit 115 determines whether the number of subject areas (subject areas of a specific object class) of the detected objects, among which the object class with the highest preset tracking priority has two or more. The tracking priority is the priority for determining the tracking target. If the above number is two or more, the gaze detection unit control unit 115 proceeds to step S303, and if the above number is one, it proceeds to step S308. Here, it is assumed that, in descending order of tracking priority, "human body", "dog", "cat", "bird", "horse", "other animals", "automobile", "motorcycle", "train", "airplane", "helicopter", "ship", "other vehicles", "fish", "insect", "reptile", "crustacean" are preset. As an example, consider a case where one subject area of the object class "dog" and two subject areas of the object class "cat" are detected. In this case, the object class with the highest tracking priority is "dog", and since the number of subject areas of the object class "dog" is one, the gaze detection unit control unit 115 proceeds to step S308.

[0056] Note that the tracking priority (the order of object classes according to the tracking priority) may be arbitrarily changed by the user. For example, the tracking priority may be changed according to the shooting mode of the imaging device 100, such as the person priority mode, the animal priority mode, the vehicle priority mode, etc. In the case of the animal priority mode, the tracking priority is changed so that the tracking of animals is prioritized. For example, in descending order of tracking priority, "dog", "cat", "bird", "horse", "other animals", " "Human body", "automobile", "motorcycle", "train", "airplane", "helicopter", "ship", "other vehicle", "fish", "insect", "reptile", "crustacean". That is, "dog", "cat", "bird", "horse", "other animal" take precedence over the "human body". In order to be able to swap the order of "dog" and "cat", it may be possible to set shooting modes such as dog priority mode and cat priority mode. The number of subject areas of the object class with the highest tracking priority may be determined from the subject detection results of one frame, or may be determined from the average of the subject detection results of multiple frames. The line-of-sight detection unit control unit 115 may proceed to step S306 when the number of subject areas of the object class with the highest tracking priority is one. In that case, the subject area of the object class with the highest tracking priority may be determined as the tracking target. The line-of-sight detection unit control unit 115 may proceed to step S307 when the number of subject areas of the object class with the highest tracking priority is two or more.

[0057] In step S303, the line-of-sight detection unit control unit 115 focuses on a plurality of subject regions of the object class with the highest following priority. The line-of-sight detection unit control unit 115 determines whether a subject region of the partial object class "pupil" is detected in only one of the plurality of subject regions. If a subject region of the partial object class "pupil" is detected in only one of the plurality of subject regions, the line-of-sight detection unit control unit 115 proceeds to step S306. In that case, the subject region of the partial object class "pupil" detected from the subject region of the object class with the highest following priority is determined as the following target. If a subject region of the partial object class "pupil" is detected in two or more of the plurality of subject regions, the line-of-sight detection unit control unit 115 proceeds to step S304. Even if a subject region of the partial object class "pupil" is not detected in any of the plurality of subject regions, the line-of-sight detection unit control unit 115 proceeds to step S304. Even when the object class with the highest following priority indicates a subject type in which no pupil is detected, the line-of-sight detection unit control unit 115 proceeds to step S304.

[0058] The partial object class is an object class indicating the type of a partial subject (a part of a subject). For example, from subject regions of object classes "human body", "dog", "cat", "bird", "horse", "other animals", subject regions of partial object classes "pupil", "head", "face" can be detected. From subject regions of object classes "fish", "insect", "reptile", "crustacean", a subject region of the partial object class "pupil" can be detected. From a subject region of the object class "motorcycle", a subject region of the partial object class "head (rider's helmet)" can be detected. From a subject region of the object class "automobile", a subject region of the partial object class "head (rider's head)" can be detected. The following priority of these partial object classes is higher than that of object classes that are not partial object classes.

[0059] Note that the line-of-sight detection unit control unit 115 may proceed to step S307 instead of proceeding to step S304. Whether or not the subject area of the partial object class "pupil" is detected may be determined from the subject detection result of one frame or from the subject detection results of a plurality of frames. For example, if the situation where the subject area of the partial object class "pupil" is detected only in the same one subject area continues over a plurality of frames, the process may proceed to step S306, and otherwise the process may proceed to step S304.

[0060] Also in step S304, the line-of-sight detection unit control unit 115 focuses on a plurality of subject areas of the object class with the highest tracking priority. The line-of-sight detection unit control unit 115 Of the plurality of subject regions, it is determined whether a subject region of the partial object class "face" and a subject region of the partial object class "head" are detected only in one subject region. If a subject region of the partial object classes "face" and "head" is detected only in one subject region among the plurality of subject regions, the gaze detection unit control unit 115 proceeds to step S306. In that case, the subject region of the partial object class "face" or "head" detected from the subject region of the object class with the highest tracking priority is determined as the tracking target. If a subject region of the partial object classes "face" and "head" is detected in each of two or more subject regions among the plurality of subject regions, the gaze detection unit control unit 115 proceeds to step S305. Even if no subject region of the partial object class "face" is detected in any of the plurality of subject regions, the gaze detection unit control unit 115 proceeds to step S305. Even if no subject region of the partial object class "head" is detected in any of the plurality of subject regions, the gaze detection unit control unit 115 proceeds to step S305. Even if the object class with the highest tracking priority indicates a subject type in which no face is detected, the gaze detection unit control unit 115 proceeds to step S305. Even if the object class with the highest tracking priority indicates a subject type in which no head is detected, the gaze detection unit control unit 115 proceeds to step S305.

[0061] Note that the line-of-sight detection unit control unit 115 may proceed to step S307 instead of proceeding to step S305. The determination in step S304 may be replaced with a determination as to whether or not a subject region of the partial object class "face" has been detected. The determination in step S304 may be replaced with a determination as to whether or not a subject region of the partial object class "head" has been detected. In that case, even if a subject region of the partial object class "face" has not been detected, a subject region of the partial object class "head" can be determined as a tracking target. The determination in step S304 may be made based on the subject detection result of one frame, or may be made based on the subject detection results of a plurality of frames. For example, if a situation where subject regions of the partial object classes "face" and "head" are detected only in the same one subject region continues over a plurality of frames, the process may proceed to step S306, and otherwise the process may proceed to step S305.

[0062] Also in step S305, the line-of-sight detection unit control unit 115 focuses on a plurality of subject regions of the object class with the highest following priority. The line-of-sight detection unit control unit 115 determines whether a subject region of a partial object class that is neither "pupil", "face", nor "head" is detected in only one of the plurality of subject regions. If a subject region of a partial object class that is neither "pupil", "face", nor "head" is detected in only one of the plurality of subject regions, the line-of-sight detection unit control unit 115 proceeds to step S306. In that case, the subject region of the partial object class detected from the subject region of the object class with the highest following priority, which is neither "pupil", "face", nor "head", is determined as the following target. If a subject region of a partial object class that is neither "pupil", "face", nor "head" is detected in two or more of the plurality of subject regions, the line-of-sight detection unit control unit 115 proceeds to step S307. Even if no subject region of a partial object class that is neither "pupil", "face", nor "head" is detected in any of the plurality of subject regions, the line-of-sight detection unit control unit 115 proceeds to step S307. Even when the object class with the highest following priority indicates a subject type in which no partial subject that is neither pupil, face, nor head is detected, the line-of-sight detection unit control unit 115 proceeds to step S307.

[0063] Note that instead of proceeding to step S307, the line-of-sight detection unit control unit 115 may proceed to step S311. Whether a subject region of a partial object class that is neither "pupil", "face", nor "head" is detected may be determined from the subject detection result of one frame or from the subject detection results of a plurality of frames. For example, if the situation where a subject region of a partial object class is detected only in the same one subject region continues over a plurality of frames, the process may proceed to step S306, and otherwise, the process may proceed to step S307.

[0064] In step S308, the gaze detection unit control unit 115 determines whether or not a subject area of the partial object class "pupil" is detected in the subject area of the object class with the highest following priority. If a subject area of the partial object class "pupil" is detected in the subject area of the object class with the highest following priority, the gaze detection unit control unit 115 proceeds to step S306. In that case, the subject area of the partial object class "pupil" detected from the subject area of the object class with the highest following priority is determined as the following target. If a subject area of the partial object class "pupil" is not detected in the subject area of the object class with the highest following priority, the gaze detection unit control unit 115 proceeds to step S309. Even when the object class with the highest following priority indicates a subject type in which no pupil is detected, the gaze detection unit control unit 115 proceeds to step S309.

[0065] Note that in step S308, similar to step S303, it may be determined whether or not a subject area of the partial object class "pupil" is detected from the subject detection result of one frame, or it may be determined from the subject detection results of a plurality of frames. Instead of proceeding to step S309, the gaze detection unit control unit 115 may proceed to step S307.

[0066] In step S309, the gaze detection unit control unit 115 determines whether a subject area of the partial object class "face" and a subject area of the partial object class "head" are detected in the subject area of the object class with the highest tracking priority. If the subject areas of the partial object classes "face" and "head" are detected in the subject area of the object class with the highest tracking priority, the gaze detection unit control unit 115 proceeds to step S306. In that case, the subject area of the partial object class "face" or "head" detected from the subject area of the object class with the highest tracking priority is determined as the tracking target. If the subject area of the partial object class "face" is not detected in the subject area of the object class with the highest tracking priority, the gaze detection unit control unit 115 proceeds to step S310. Even if the subject area of the partial object class "head" is not detected in the subject area of the object class with the highest tracking priority, the gaze detection unit control unit 115 proceeds to step S310. Even if the object class with the highest tracking priority indicates a subject type in which a face is never detected, the gaze detection unit control unit 115 proceeds to step S310. Even if the object class with the highest tracking priority indicates a subject type in which a head is never detected, the gaze detection unit control unit 115 proceeds to step S310.

[0067] Note that, similar to the determination in step S304, the determination in step S309 may be replaced with a determination of whether a subject area of the partial object class "face" is detected. The determination in step S309 may be replaced with a determination of whether a subject area of the partial object class "head" is detected. The determination in step S309 may be made based on the subject detection result of one frame or may be made based on the subject detection results of a plurality of frames. Instead of proceeding to step S310, the gaze detection unit control unit 115 may proceed to step S307.

[0068] In step S310, the line-of-sight detection unit control unit 115 determines whether a subject region of a partial object class that is neither a "pupil", nor a "face", nor a "head" is detected in the subject region of the object class with the highest tracking priority. If a subject region of a partial object class that is neither a "pupil", nor a "face", nor a "head" is detected in the subject region of the object class with the highest tracking priority, the line-of-sight detection unit control unit 115 proceeds to step S306. In that case, the subject region of the partial object class that is neither a "pupil", nor a "face", nor a "head" and is detected from the subject region of the object class with the highest tracking priority is determined as the tracking target. If a subject region of a partial object class that is neither a "pupil", nor a "face", nor a "head" is not detected in the subject region of the object class with the highest tracking priority, the line-of-sight detection unit control unit 115 proceeds to step S311. Even when the object class with the highest tracking priority indicates a subject type in which a partial subject that is neither a pupil, nor a face, nor a head is not detected, the line-of-sight detection unit control unit 115 proceeds to step S311.

[0069] Note that, similar to the determination in step S305, the determination in step S310 may be made based on the subject detection result of one frame or may be made based on the subject detection results of a plurality of frames. Instead of proceeding to step S311, the line-of-sight detection unit control unit 115 may proceed to step S307.

[0070] In step S312, the line-of-sight detection unit control unit 115 calculates the total number of detected subject regions for each object class (that is, the total number of detected subject regions), and determines whether the calculated number is 1. If the calculated number is 1, the line-of-sight detection unit control unit 115 proceeds to step S311. If the calculated number is 2 or more or 0 (zero), the line-of-sight detection unit control unit 115 proceeds to step S307.

[0071] Note that when the number calculated is two or more, the line-of-sight detection unit control unit 115 may proceed to step S311. The total number of subject regions may be determined from the subject detection result of one frame, or may be determined from the average of the subject detection results of a plurality of frames.

[0072] In step S311, the line-of-sight detection unit control unit 115 focuses on one subject region. When proceeding from step S310 to step S311, the line-of-sight detection unit control unit 115 focuses on a subject region of a specific object class. When proceeding from step S312 to step S311, the line-of-sight detection unit control unit 115 focuses on a subject region of an object class that is not a specific object class. The line-of-sight detection unit control unit 115 determines whether or not the size of the focused subject region is equal to or greater than a predetermined threshold. When the size of the focused subject region is equal to or greater than the predetermined threshold, the line-of-sight detection unit control unit 115 proceeds to step S307, and when the size of the focused subject region is less than the predetermined threshold, the line-of-sight detection unit control unit 115 proceeds to step S306. When the process proceeds to step S306, the subject region focused on in step S311 is determined to be the tracking target.

[0073] Note that as described above, in step S305, instead of proceeding to step S307, the line-of-sight detection unit control unit 115 may proceed to step S311. In that case, the line-of-sight detection unit control unit 115 focuses on any one of a plurality of subject regions of a specific object class. Also, in step S312, when the total number of detected subject regions is two or more, the line-of-sight detection unit control unit 115 may proceed to step S311. In that case, the line-of-sight detection unit control unit 115 focuses on any one of the plurality of detected subject regions. The method for selecting the subject region to be focused on is not particularly limited. For example, any one of the plurality of subject regions may be randomly selected, or any one of the plurality of subject regions may be selected based on parameters such as the size and position of each subject region. Whether or not the size of the focused subject region is equal to or greater than a predetermined threshold is determined based on the subject in one frame It may be determined based on the body detection result, or it may be determined based on the subject detection results of a plurality of frames. For example, if the situation where the size of the subject area of interest is less than a predetermined threshold continues over a plurality of frames, the process proceeds to step S306; otherwise, the process may proceed to step S307.

[0074] Also, different thresholds may be predetermined for each object class as the threshold for comparing with the size of the subject area of interest. As an example, consider a case where the number of horizontal pixels in the image (captured image) stored in the tracking memory 110 is 160 pixels and the number of vertical pixels is 120 pixels. In this case, a size of 40 pixels in the horizontal direction × 30 pixels in the vertical direction may be determined as the threshold for comparing with the size of the subject area of the object class "train". A size of 80 pixels in the horizontal direction × 60 pixels in the vertical direction may be determined as the threshold for comparing with the size of the subject area of the object class "flower".

[0075] In step S306, the gaze detection unit control unit 115 controls to disable the operation of the gaze detection unit so that gaze detection (the operation of detecting the user's gaze) is not performed. For example, the gaze detection unit control unit 115 controls to disable the operation of at least any one of the first infrared light irradiation unit 111, the second imaging element 120, the third preprocessing unit 121, and the second detection unit 126. Thereby, the power consumption of the imaging device 100 can be reduced. Since the process of step S306 is performed when it is not necessary to use the fixation point position for determining the AF point (it is sufficient to use the tracking target), there is no problem even if the operation of the gaze detection unit is disabled. As an example, consider a case where the process proceeds from step S303 to step S306, that is, among the plurality of subject areas of the object class with the highest tracking priority, the subject area of the partial object class "pupil" is detected only in one subject area. In this case, the central control unit 122 determines the position (tracking result) of the subject area of the partial object class "pupil" as the AF point. Therefore, even if the operation of the gaze detection unit is disabled, autofocus on the subject can be suitably performed (with normal performance).

[0076] In step S307, the gaze detection unit control unit 115 effectively controls the operation of the gaze detection unit so that gaze detection is performed. For example, the gaze detection unit control unit 115 controls the operation cycles of the first infrared light irradiation unit 111, the second imaging element 120, the third preprocessing unit 121, and the second detection unit 126 to a predetermined operation cycle. Since the process of step S307 is performed when the fixation point position is used to determine the AF point, it is necessary to enable the operation of the gaze detection unit. As an example, consider the case where the process proceeds from step S311 to step S307, that is, when the size of the subject area of interest is equal to or greater than a predetermined threshold. In this case, if the fixation point position exists within the subject area of interest, the central control unit 122 determines the fixation point position as the AF point. Therefore, although the power consumption of the imaging device 100 increases, the gaze detection unit is enabled so that autofocus can be suitably performed.

[0077] Note that the process of step S306 may be performed when the possibility of using the fixation point position is low. The process of step S306 may be performed when the fixation point position is not used to determine the AF point but is used for other processes. When the process of step S306 is performed, the gaze detection unit may be used as a general-purpose input device. For example, when the process of step S306 is performed, icons of different shooting modes may be displayed on the display unit 118, and the change of the shooting mode may be determined based on the fixation point position and the fixation time. In such a case, the gaze detection unit cannot be disabled. Therefore, in step S306, the gaze detection unit control unit 115 may control the operation cycle of the gaze detection unit to an operation cycle longer than the operation cycle determined in step S307. For example, when an operation cycle of 1 / 60 second (an operation cycle of performing 60 operations per second) is determined in step S307, an operation cycle of 1 / 30 second (an operation cycle of performing 30 operations per second) may be determined in step S306. When an operation cycle of 1 / 120 second (per second When determining the operation period (for performing 120 operations), in step S306, an operation period of 1 / 60 second or 1 / 30 second may be determined. The line-of-sight detection unit control unit 115 controls the operation period of at least any one of the first infrared light irradiation unit 111, the second imaging element 120, the third preprocessing unit 121, and the second detection unit 126 to be longer than the operation period determined in step S307. By increasing the operation period, the execution frequency of the operation (the number of times the operation is executed per unit time) decreases, so that the power consumption of the imaging device 100 can be reduced.

[0078] Also, the line-of-sight detection unit control unit 115 may control one operation among the first infrared light irradiation unit 111, the second imaging element 120, the third preprocessing unit 121, and the second detection unit 126, or may control two or more (part or all) operations. When controlling the operation periods of two or more of the first infrared light irradiation unit 111, the second imaging element 120, the third preprocessing unit 121, and the second detection unit 126, their operation periods may be controlled to be the same operation period, or may be controlled to be different operation periods.

[0079] Note that the method for controlling the operation of the gaze detection unit is not limited to the above method. In step S306 or step S307, the gaze detection unit control unit 115 may vary the operation cycle of the gaze detection unit according to the subject detection result. As an example, consider a case where three or more subject areas of the object class "human body" are detected and a subject area of an object class related to sports such as "ball" or "racket" is detected. In this case, the gaze detection unit control unit 115 may determine that the user is taking a sports photo and determine the shortest operation cycle such as 1 / 240 second as the operation cycle of the gaze detection unit. Consider a case where one subject area of the object class "cooking" is detected and the size of the subject area is equal to or greater than a predetermined threshold. In this case, the gaze detection unit control unit 115 may determine that the user is taking a cooking photo and determine a long operation cycle such as 1 / 15 second as the operation cycle of the gaze detection unit. Consider a case where one subject area of the object class "train" is detected and the size of the subject area is equal to or greater than a predetermined threshold. In this case, the gaze detection unit control unit 115 may determine that the user is taking a train photo and determine a short operation cycle such as 1 / 60 second as the operation cycle of the gaze detection unit.

[0080] In addition, when the user is shooting sports, if only one subject area of the object class "human body" is detected, it is highly likely that the subject area is the area of the athlete (player) on which the user wants to focus. Therefore, if only one subject area of the object class "human body" is detected, the position of the subject area of the object class "human body" may be determined as the AF point without using the fixation point position. And when the user is shooting sports, the gaze detection unit control unit 115 may control the operation of the gaze detection unit according to the number of subject areas of the object class "human body". Here, consider the case where only one subject area of the object class "human body" is detected and a subject area of an object class related to sports such as "ball" or "racket" is detected. In this case, the position of the subject area of the object class "human body" may be determined as the AF point without using the fixation point position. Therefore, the gaze detection unit control unit 115 may disable the operation of the gaze detection unit or may increase the operation cycle of the gaze detection unit. Consider the case where a plurality of subject areas of the object class "human body" are detected and a subject area of an object class related to sports is detected. In this case, it is difficult for the user to easily determine which of the plurality of subject areas of the object class "human body" the user wants to focus on. Therefore, the gaze detection unit control unit 115 may enable the operation of the gaze detection unit or may shorten the operation cycle of the gaze detection unit.

[0081] FIG. 4 shows an example of an image captured by the first imaging device 102 and displayed on the display unit 118, the fixation point position, the AF point, and the like. Using FIG. 4, a specific example of the operation of FIG. 3 will be described. Hereinafter, the case where the processes of step S306 and step S307 are switched and executed according to the size of the subject area will be described. FIG. 4 shows an example of an image captured by the first imaging device 102 and displayed on the display unit 118, the fixation point position, the AF point, and the like. Using FIG. 4, a specific example of the operation of FIG. 3 will be described. Hereinafter, the case where the processes of step S306 and step S307 are switched and executed according to the size of the subject area will be described.

[0082] The image 400 is an image captured by the first imaging device 102 and displayed on the display unit 118. A potted flower 401 is captured in the image 400. The first detection unit 112 detects a subject region 404 circumscribing the potted flower 401 from the image 400 stored in the tracking memory 110. Then, as information regarding the subject region 404, the first detection unit 112 generates an object class "flower" and the size of the subject region 404 (40 pixels in the horizontal direction × 40 pixels in the vertical direction). Since the object class "flower" is not a specific object class (a predefined tracking candidate), the process proceeds from step S301 to step S312. Also, since the total number of detected subject regions is one, the process proceeds from step S312 to step S311. Here, the threshold value (the threshold value for comparing with the size of the subject region) used in step S311 is set to a size of 80 pixels in the horizontal direction × 60 pixels in the vertical direction. Since the size of the subject region 404 (40 pixels in the horizontal direction × 40 pixels in the vertical direction) is less than the threshold value, the process proceeds to step S306 and the operation of the gaze detection unit is disabled. Since the operation of the gaze detection unit is disabled, the central control unit 122 determines the center position 403 of the subject region 404, instead of the gaze point position 402, as the AF point. Note that since the operation of the gaze detection unit is disabled, the gaze point position 402 is neither detected nor displayed.

[0083] Image 410 is an image that the display unit 118 displays in a frame (timing) different from that of image 400. A potted flower 411 is captured in image 410. The first detection unit 112 detects a subject region 414 circumscribing the potted flower 411 from the image 410 stored in the tracking memory 110. Then, as information regarding the subject region 414, the first detection unit 112 generates an object class "flower" and the size of the subject region 414 (90 pixels in the horizontal direction × 90 pixels in the vertical direction). Similar to the case of image 400, the process proceeds to step S311. Since the size of the subject region 414 (90 pixels in the horizontal direction × 90 pixels in the vertical direction) is equal to or greater than the threshold value (80 pixels in the horizontal direction × 60 pixels in the vertical direction), the process proceeds to step S307, and the operation of the gaze detection unit becomes effective. Since the operation of the gaze detection unit has become effective, the central control unit 122 determines the same position 413 as the fixation point position 412, instead of the center position of the subject region 414, as the AF point.

[0084] In this way, when the size of the subject region is equal to or greater than a predetermined threshold value, by enabling the gaze detection unit and determining the fixation point position as the AF point, autofocus can be performed with the AF point at the position as intended by the user. Also, when the size of the subject region is less than the predetermined threshold value, by disabling the gaze detection unit, the power consumption of the imaging device 100 can be reduced. If the subject region (subject) is small, in many cases, no matter which position within the subject region is determined as the AF point, substantially the same image is captured after autofocus. Therefore, even if the fixation point position is not determined as the AF point (even if the center position of the subject region is determined as the AF point), autofocus close to the user's intention can be performed.

[0085] As described above, according to Embodiment 1, based on the subject detection result, the operation of the gaze detection unit is controlled to be effective or ineffective, or the operation cycle of the gaze detection unit is controlled. Thereby, gaze detection can be enabled or accelerated only when necessary. In other words, when it is not necessary, gaze detection can be disabled or decelerated. Thereby, while suppressing a decrease in the accuracy of processing using gaze detection, the power consumption of the imaging device 100 can be reduced. The first infrared light irradiation unit 111 of the gaze detection unit emits light, so the power consumption is relatively large, and the second imaging element 120 of the gaze detection unit performs imaging, so the power consumption is relatively large. Therefore, as an effect of reducing the power consumption of the gaze detection unit, a high effect can be expected.

[0086] In addition, when gaze detection is necessary, for example, the following cases are included. · When a fast-moving subject (tracking candidate) is detected · When the size of the subject is large and an AF point should be selected inside the subject · When a combination of subjects such as a person and a ball, which can be determined that the user is (or intends to) shoot sports, is detected · When a plurality of subjects are detected · When no subject is detected

[0087] When gaze detection is not required, for example, the following cases are included. · When only one specific type of subject such as a pupil or a biker's helmet is detected and the position to be determined as the AF point is clear · When the size of the subject is small and the position to be determined as the AF point is clear

[0088] Also, according to Embodiment 1, when photographing a train, the following operations are possible. When the train approaches from a distance, since the size of the train in the captured image is less than the threshold value, the gaze detection is disabled, and the center position of the train is determined as the AF point. When the train approaches and the size of the train in the image becomes equal to or greater than the threshold value, the gaze detection is enabled, and the fixation point position is determined as the AF point. Then, when the train moves away and the size of the train in the image becomes less than the threshold value, the gaze detection is disabled, and the center position of the train is determined as the AF point.

[0089] <Embodiment 2> Hereinafter, Embodiment 2 of the present invention will be described. In the following, the description of the same points (configurations, processes, etc.) as those in Embodiment 1 will be omitted, and the points different from those in Embodiment 1 will be described. In Embodiment 2, the gaze detection unit control unit 115 controls the operation of the gaze detection unit based on the subject detection result only when the imaging device 100 is set to a specific mode.

[0090] FIG. 5 is a flowchart showing an example of the operation of the imaging device 100 according to Embodiment 2. FIG. 5 shows an example of the operation of controlling the gaze detection unit based on the shooting mode of the imaging device 100.

[0091] In step S501, the gaze detection unit control unit 115 determines whether the shooting mode of the imaging device 100 is the AF mode for non-moving object shooting. If the shooting mode of the imaging device 100 is not the AF mode for non-moving object shooting, the gaze detection unit control unit 115 proceeds to step S502, and if the shooting mode of the imaging device 100 is the AF mode for non-moving object shooting, the gaze detection unit control unit 115 proceeds to step S505.

[0092] In step S502, the line-of-sight detection unit control unit 115 determines whether the shooting mode of the imaging device 100 is the AF mode for moving object shooting. If the shooting mode of the imaging device 100 is not the AF mode for moving object shooting, the line-of-sight detection unit control unit 115 proceeds to step S503, and if the shooting mode of the imaging device 100 is the AF mode for moving object shooting, the line-of-sight detection unit control unit 115 proceeds to step S504. In Embodiment 2, it is assumed that as the AF mode, an AF mode for non-moving object shooting, an AF mode for moving object shooting, or an automatic discrimination AF mode is set. Therefore, the case where the process proceeds to step S503 means that the shooting mode of the imaging device 100 is the automatic discrimination AF mode. The automatic discrimination AF mode is an AF mode that automatically discriminates between moving objects and non-moving objects and performs shooting.

[0093] In step S503, based on the subject detection result, the line-of-sight detection unit control unit 115 controls the operation of the line-of-sight detection unit. For example, the line-of-sight detection unit control unit 115 controls the line-of-sight detection unit with the operation shown in FIG. 3.

[0094] In step S504, in order to shoot a moving object, the line-of-sight detection unit control unit 115 enables the operation of the line-of-sight detection unit and fixes the operation cycle of the line-of-sight detection unit at the shortest operation cycle (for example, 1 / 240 second). Based on the subject detection result, the operation of the line-of-sight detection unit is not controlled.

[0095] In step S505, in order to shoot a non-moving object, the line-of-sight detection unit control unit 115 enables the operation of the line-of-sight detection unit and fixes the operation cycle of the line-of-sight detection unit at the longest operation cycle (for example, 1 / 5 second). Based on the subject detection result, the operation of the line-of-sight detection unit is not controlled.

[0096] As described above, according to Embodiment 2, only in a specific mode, the operation of the line-of-sight detection unit is controlled based on the subject detection result. By doing so, the power consumption of the imaging device 100 can be reduced and the convenience of the imaging device 100 can be improved.

[0097] <Embodiment 3> Hereinafter, Embodiment 3 of the present invention will be described. In the following, the description of the same points (configurations, processes, etc.) as those in Embodiment 1 will be omitted, and the points different from those in Embodiment 1 will be described. The infrared light irradiated by the first infrared light irradiation unit 111 for gaze detection may reduce the accuracy of eye contact detection, and the infrared light irradiated by the second infrared light irradiation unit 129 for eye contact detection may reduce the accuracy of gaze detection. For example, when the infrared light irradiated by the first infrared light irradiation unit 111 and the infrared light irradiated by the second infrared light irradiation unit 129 interfere with each other, the accuracy of gaze detection and eye contact detection decreases. Therefore, in Embodiment 3, the state (operation) of the eye contact detection unit 128 is controlled according to the state of the gaze detection unit.

[0098] FIG. 6 is a flowchart showing an example of the operation of the imaging device 100 according to Embodiment 3. FIG. 6 shows an example of an operation of controlling the state (operation) of the eye contact detection unit 128 according to the state of the gaze detection unit.

[0099] In step S601, the gaze detection unit control unit 115 determines whether the operations of the gaze detection unit (the first infrared light irradiation unit 111, the second imaging element 120, etc.) are valid. If the operations of the gaze detection unit are not valid (invalid), the process proceeds to step S602, and if the operations of the gaze detection unit are valid, the process proceeds to step S603.

[0100] In step S602, the line-of-sight detection unit control unit 115 determines that it can control the operation of the eye detection unit 128 (the second infrared light irradiation unit 129 and the infrared light sensing unit 131) in any way because the operation of the line-of-sight detection unit is not effective. For example, since the infrared light emitted from the second infrared light irradiation unit 129 is not imaged by the second imaging element 120, the line-of-sight detection unit control unit 115 determines that the second infrared light irradiation unit 129 may emit infrared light. Also, since the infrared light emitted from the first infrared light irradiation unit 111 does not enter the infrared light sensing unit 131, the line-of-sight detection unit control unit 115 determines that the infrared light sensing unit 131 may sense the incident infrared light. Then, the line-of-sight detection unit control unit 115 enables the operation of the eye detection unit 128. Since the shorter the operation cycle of the eye detection unit 128, the better the responsiveness to detect eye contact, it is preferable for the line-of-sight detection unit control unit 115 to shorten the operation cycle of the eye detection unit 128. For example, when the shortest operation cycle of the eye detection unit 128 is 1 / 120 second, the line-of-sight detection unit control unit 115 may control the operation cycle of the eye detection unit 128 to 1 / 120 second.

[0101] In step S603, the line-of-sight detection unit control unit 115 determines whether the operation cycle of the line-of-sight detection unit is a predetermined operation cycle (the first operation cycle). For example, the first operation cycle is the operation cycle when the operation in FIG. 3 is not performed, and by the operation in FIG. 3, the operation cycle of the line-of-sight detection unit may be controlled to an operation cycle longer or shorter than the first operation cycle. When the operation cycle of the line-of-sight detection unit is the first operation cycle, the line-of-sight detection unit control unit 115 proceeds to step S604, and when the operation cycle of the line-of-sight detection unit is not the first operation cycle, the line-of-sight detection unit control unit 115 proceeds to step S605.

[0102] In step S604, the line-of-sight detection unit control unit 115 controls the operation cycle of the eye detection unit 128 to a predetermined operation cycle (the second operation cycle). The second operation cycle, which is the operation cycle of the eye detection unit 128, is an operation cycle determined based on the first operation cycle, which is the operation cycle of the line-of-sight detection unit.

[0103] FIG. 7 shows an example of the operation of the first infrared light irradiation unit 111 in the first operation cycle and the operation of the second infrared light irradiation unit 129 in the second operation cycle. The time from timing T701 to timing T703 and the time from timing T703 to timing T705 are each 1 / 60 second. The time from timing T705 to timing T707 and the time from timing T707 to timing T709 are also each 1 / 60 second.

[0104] Here, the first infrared light irradiation unit 111 operates as follows. In the following operation, the first infrared light irradiation unit 111 is turned on (infrared light irradiation) and off (non-infrared light irradiation) at a frequency of once every 1 / 30 second. That is, the operation cycle of the first infrared light irradiation unit 111 is controlled to be 1 / 30 second (the first operation cycle). · Irradiate infrared light from timing T701 to timing T702 (infrared light irradiation 710). · Do not irradiate infrared light from timing T702 to timing T705 (non-infrared light irradiation 711). · Irradiate infrared light from timing T705 to timing T706 (infrared light irradiation 712). · Do not irradiate infrared light from timing T706 to timing T709 (non-infrared light irradiation 713).

[0105] The second infrared light irradiation unit 129 operates as follows according to the above operation of the first infrared light irradiation unit 111. In the following operation, the second infrared light irradiation unit 129 is turned on (infrared light irradiation) and off (non-infrared light irradiation) at a frequency of once every 1 / 30 second. That is, the operation cycle of the second infrared light irradiation unit 129 is also controlled to be 1 / 30 second (the second operation cycle). · Do not irradiate infrared light from timing T701 to timing T703 (non-infrared light irradiation 720). · Irradiate infrared light from timing T703 to timing T704 (infrared light irradiation 721). · Do not irradiate infrared light from timing T704 to timing T707 (non-infrared light irradiation 722). · Infrared light is irradiated from timing T707 to timing T708 (infrared light irradiation 723). · Infrared light is not irradiated from timing T708 to timing T709 (non-infrared light irradiation 724).

[0106] As described above, in the example of FIG. 7, both the first infrared light irradiation unit 111 and the second infrared light irradiation unit 129 operate with an operation cycle of 1 / 30 second. However, so that the period during which both the first infrared light irradiation unit 111 and the second infrared light irradiation unit 129 are lit becomes short (does not occur), the periods of infrared light irradiations 721 and 723 of the second infrared light irradiation unit 129 are shifted from the periods of infrared light irradiations 710 and 712 of the first infrared light irradiation unit 111. By shifting the period of infrared light irradiation of the second infrared light irradiation unit 129 from the period of infrared light irradiation of the first infrared light irradiation unit 111, it is possible to reduce the influence of the infrared light of the first infrared light irradiation unit 111 on eye detection and the influence of the infrared light of the second infrared light irradiation unit 129 on gaze detection. Note that FIG. 7 is an example, and the first infrared light irradiation unit 111 and the second infrared light irradiation unit 129 may be controlled so that the period of infrared light irradiation of the first infrared light irradiation unit 111 and the period of infrared light irradiation of the second infrared light irradiation unit 129 are shifted from each other. The first operation cycle may be longer or shorter than 1 / 30 second, the second operation cycle may be longer or shorter than 1 / 30 second, the second operation cycle may be the same as the first operation cycle, or the second operation cycle may be different from the first operation cycle. For example, even if the responsiveness of eye detection is allowed to decrease, the operation cycle (second operation cycle) of the eye detection unit 128 may be controlled to 1 / 15 fps so that the power consumption of the eye detection unit 128 is reduced. When the operation cycle of the eye detection unit 128 is lengthened, the time of one infrared light irradiation does not change, and it is assumed that the time of one non-infrared light irradiation becomes longer. Therefore, when the operation cycle of the gaze detection unit is constant, the longer the operation cycle of the eye detection unit 128, the easier it is to shift the period of infrared light irradiation of the second infrared light irradiation unit 129 from the period of infrared light irradiation of the first infrared light irradiation unit 111, and it is easier to suppress a decrease in the accuracy of gaze detection and eye detection.

[0107]

[0108] ​In step S605, the gaze detection unit control unit 115 determines whether the operation cycle of the gaze detection unit is controlled to be longer than the first operation cycle. If the operation cycle of the gaze detection unit is controlled to be longer than the first operation cycle, the gaze detection unit control unit 115 proceeds to step S606. If the operation cycle of the gaze detection unit is not controlled to be longer than the first operation cycle (specifically, it is controlled to be shorter than the first operation cycle), the gaze detection unit control unit 115 proceeds to step S607.

[0109] In step S606, since the operation cycle of the gaze detection unit is longer than the first operation cycle, the gaze detection unit control unit 115 makes the operation cycle of the eye contact detection unit 128 shorter than the second operation cycle. The state where the operation cycle of the first infrared light irradiation unit 111 is long means that the time during which the first infrared light irradiation unit 111 does not irradiate infrared light (the time between infrared light irradiations) is long. In this case, it is easy to shift the infrared light irradiation period of the second infrared light irradiation unit 129 from the infrared light irradiation period of the first infrared light irradiation unit 111. Therefore, the gaze detection unit control unit 115 increases the responsiveness of eye contact detection by making the operation cycle of the eye contact detection unit 128 shorter than the second operation cycle.

[0110] In step S607, since the operation cycle of the gaze detection unit is shorter than the first operation cycle, the gaze detection unit control unit 115 makes the operation cycle of the eye contact detection unit 128 longer than the second operation cycle. The state where the operation cycle of the first infrared light irradiation unit 111 is short means that the time during which the first infrared light irradiation unit 111 does not irradiate infrared light (the time between infrared light irradiations) is short. In this case, it is difficult to shift the infrared light irradiation period of the second infrared light irradiation unit 129 from the infrared light irradiation period of the first infrared light irradiation unit 111. Therefore, the gaze detection unit control unit 115 makes the operation cycle of the eye contact detection unit 128 longer than the second operation cycle to shift the infrared light irradiation period of the second infrared light irradiation unit 129 from the infrared light irradiation period of the first infrared light irradiation unit 111.

[0111] As described above, according to Embodiment 3, the state (operation) of the eye contact detection unit 128 is controlled according to the state of the line-of-sight detection unit. By doing so, it is possible to reduce the influence of the infrared light of the first infrared light irradiation unit 111 on the eye contact detection and the influence of the infrared light of the second infrared light irradiation unit 129 on the line-of-sight detection, and suppress the deterioration of the accuracy of the line-of-sight detection and the eye contact detection.

[0112] <Embodiment 4> Hereinafter, Embodiment 4 of the present invention will be described. In the following, the description of the same points (configuration, processing, etc.) as those in Embodiment 1 will be omitted, and the points different from those in Embodiment 1 will be described. In Embodiment 4, the operation of the line-of-sight detection unit is controlled based on the remaining battery level of the imaging device.

[0113] FIG. 8 is a block diagram schematically showing an example of the internal structure of an imaging device 800 according to Embodiment 4. The imaging device 800 has a configuration in which a battery 802 and a remaining battery level determination unit 801 are added to the imaging device 100 shown in FIG. 1. The battery 802 supplies power to each component of the imaging device 800 to drive each component. The remaining battery level determination unit 801 determines whether the imaging device 800 (battery 802) is in a low battery state, and outputs the determination result (remaining battery level determination result) to the line-of-sight detection unit control unit 115. The low battery state is a state where the remaining battery level of the battery 802 is equal to or lower than a predetermined threshold value. The line-of-sight detection unit control unit 115 controls the operation of the line-of-sight detection unit based on the remaining battery level determination result.

[0114] FIG. 9 is a flowchart showing an example of the operation of the imaging device 800. FIG. 9 shows an example of the operation of controlling the line-of-sight detection unit based on the remaining battery level determination result. At the start of the operation in FIG. 9, it is assumed that the line-of-sight detection unit is operating with a predetermined operation cycle.

[0115] In step S901, the line-of-sight detection unit control unit 115 determines whether the battery remaining amount determination result obtained from the battery remaining amount determination unit 801 indicates a low battery state. If the battery remaining amount determination result does not indicate a low battery state, the line-of-sight detection unit control unit 115 proceeds to step S902; if the battery remaining amount determination result indicates a low battery state, the line-of-sight detection unit control unit 115 proceeds to step S903.

[0116] In step S902, since the battery 802 is not in a low battery state, the line-of-sight detection unit control unit 115 does not control the operation cycle of the line-of-sight detection unit to remain at the predetermined operation cycle.

[0117] In step S903, since the battery 802 is in a low battery state, the line-of-sight detection unit control unit 115 controls the operation of the line-of-sight detection unit. For example, the line-of-sight detection unit control unit 115 controls the line-of-sight detection unit in the operation shown in FIG. 3.

[0118] Note that the operations shown in FIGS. 5 and 6 may be combined with the operation shown in FIG. 9. For example, the operation shown in FIG. 5 may not be performed if it is not a low battery state, and the operation shown in FIG. 5 may be performed if it is a low battery state. The operation shown in FIG. 5 may be performed if it is not a low battery state, and the operation shown in FIG. 3 may be performed if it is a low battery state.

[0119] Also, instead of switching whether to control the operation of the line-of-sight detection unit according to whether it is a low battery state, the control method of the operation of the line-of-sight detection unit may be switched. For example, the line-of-sight detection unit may be controlled in the operation shown in FIG. 3 regardless of whether it is a low battery state, but if it is not a low battery state, the process of step S311 in FIG. 3 may be omitted and the process may proceed to step S307.

[0120] As described above, according to the fourth embodiment, the operation of the gaze detection unit is controlled based on the determination result of whether it is in the low battery state. For example, if it is not in the low battery state, the operation cycle of the gaze detection unit is controlled to a predetermined operation cycle. If it is in the low battery state, the operation of the gaze detection unit is controlled based on the subject detection result. By doing so, when the remaining battery level is low, the operation of the gaze detection unit can be invalidated or slowed down based on the subject detection result to reduce the power consumption of the imaging device 800, thereby extending the shooting time of the imaging device 800. Also, when the remaining battery level is high, the operability of various operations using the gaze can be improved by operating the gaze detection unit at a high speed with a short operation cycle. When the remaining battery level is low, the operation of the gaze detection unit can be invalidated or slowed down based on the subject detection result to reduce the power consumption of the imaging device 800, thereby extending the shooting time of the imaging device 800. Also, when the remaining battery level is high, the operability of various operations using the gaze can be improved by operating the gaze detection unit at a high speed with a short operation cycle.

[0121] Note that the first to fourth embodiments are merely examples, and configurations obtained by appropriately modifying or changing the configurations of the first to fourth embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the first to fourth embodiments are also included in the present invention.

[0122] For example, although an example of applying the present invention to an imaging device having a structure such as a mirrorless single-lens camera has been described, the imaging device to which the present invention can be applied is not limited to a mirrorless single-lens camera, and may be a single-lens reflex camera or the like. That is, the user may look through the viewfinder and view the optical image of the subject itself rather than the image displayed on the display unit.

[0123] The imaging device of the present invention may be a smartphone, a tablet terminal, or the like. For example, a smartphone performs imaging with an out-camera provided on the back of the smartphone, and displays the image captured by the out-camera on a display unit provided on the front of the smartphone. Also, the smartphone detects the user's gaze using an in-camera provided on the front of the smartphone. Then, the smartphone detects a subject from the image captured by the out-camera, and controls the operation of the in-camera or the like based on the detection result.

[0124] The imaging device of the present invention may be a glasses (goggle) type wearable terminal. For example, a glasses type wearable terminal is provided with a camera for imaging the visual field range of a user wearing the terminal and a gaze detection sensor for detecting the gaze of the user. Then, the glasses type wearable terminal detects a subject from an image captured by the camera and controls the operation of the gaze detection sensor and the like based on the detection result. In such a wearable terminal, for example, AR (Augmented Reality) technology is used. That is, the present invention is applicable to electronic devices (such as AR glasses) that utilize AR technology. Similarly, the present invention is also applicable to electronic devices that utilize other xR technologies such as VR (Virtual Reality) technology, MR (Mixed Reality) technology, and SR (Substitutional Reality) technology.

[0125] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of Reference Numerals

[0126] 100, 800: Imaging device 102: First imaging element 112: First detection unit 115: Gaze detection unit control unit 111: First infrared light irradiation unit 120: Second imaging element 121: Third preprocessing unit 126: Second detection unit

Claims

1. An imaging means, a line-of-sight detection means for detecting the line of sight of a user, a subject detection means for detecting a subject from an image captured by the imaging means, and a control means for changing the state of the line-of-sight detection means to a second state in which the power consumption of the line-of-sight detection means is lower than that in the first state when a specific type of subject is detected by the subject detection means while the state of the line-of-sight detection means is the first state for detecting the line of sight. An imaging device characterized by comprising the above.

2. The line-of-sight detection means includes an irradiation means for irradiating the user with infrared light, a second imaging means for imaging the user, and a detection means for detecting the line of sight of the user based on an image captured by the second imaging means. The imaging device according to claim 1, characterized by the above.

3. The specific type of subject is a subject to be tracked. The imaging device according to claim 1 or 2, characterized by the above.

4. The specific type of subject is a subject that may move. The imaging device according to any one of claims 1 to 3, characterized by the above.

5. The specific type of subject includes at least any one of a creature belonging to mammals, a creature belonging to birds, a creature belonging to reptiles, a creature belonging to fish, a creature belonging to crustaceans, an insect, and a vehicle. The imaging device according to any one of claims 1 to 4, characterized by the above.

6. The specific type of subject includes at least any one of a human body, a cat, a dog, and a horse. The imaging device according to any one of claims 1 to 5, characterized by the above.

7. The specific type of subject includes at least any one of a railway vehicle, an automobile, an aircraft, and a ship. The imaging device according to any one of claims 1 to 6, characterized by the above.

8. The second state is a state in which the operation of the line-of-sight detection means is stopped. The imaging device according to any one of claims 1 to 7, characterized by the above.

9. The second state is a state in which the operation period of the line-of-sight detection means is longer than that in the first state. The imaging device according to any one of claims 1 to 7, characterized by the above.

10. The subject detection means detects a subject and a partial subject that is a part of the subject from the image. The imaging device according to any one of claims 1 to 9, characterized by the above.

11. When the state of the line-of-sight detection means is the first state, even if the subject detection means detects the specific type of subject, if the partial subject of the subject is not detected, the control means does not change the state of the line-of-sight detection means The imaging device according to claim 10, characterized in that

12. When the state of the line-of-sight detection means is the first state and the subject detection means detects the specific type of subject, even if the partial subject of the subject is not detected by the subject detection means, if the size of the subject is smaller than the threshold value, the control means changes the state of the line-of-sight detection means to the second state The imaging device according to claim 11, characterized in that

13. A plurality of types with different priorities are predetermined as the specific types When the state of the line-of-sight detection means is the first state and the subject detection means detects a plurality of specific types of subjects, based on the detection result of the subject detection means for the specific type with the highest priority among the plurality of specific types, the control means changes the state of the line-of-sight detection means The imaging device according to claim 11 or 12, characterized in that

14. When the state of the line-of-sight detection means is the first state, even if the subject detection means detects the specific type of subject, if a plurality of subjects of the same type are detected as the specific type of subject and a plurality of partial subjects of the same type corresponding to the plurality of subjects are detected, the control means does not change the state of the line-of-sight detection means The imaging device according to any one of claims 10 to 13, characterized in that

15. When the state of the line-of-sight detection means is the first state and the subject detection means detects a plurality of subjects of the same type as the specific type of subject if a partial subject of the first type is detected only in any one of the plurality of subjects, the control means changes the state of the line-of-sight detection means to the second state if a partial subject of the first type is detected in two or more of the plurality of subjects and a partial subject of the second type is detected only in any one of them, the control means changes the state of the line-of-sight detection means to the second state Among the plurality of subjects, even if the first type of partial subject is detected in two or more subjects and the second type of partial subject is detected in two or more subjects, if the third type of partial subject is detected only in either of them, the state of the line-of-sight detection means is changed to the second state. Among the plurality of subjects, if the first type of partial subject is detected in two or more subjects, the second type of partial subject is detected in two or more subjects, and the third type of partial subject is detected in two or more subjects, the state of the line-of-sight detection means is not changed. The imaging device according to any one of claims 10 to 14, characterized by the above.

16. The partial subject includes a pupil. The imaging device according to any one of claims 10 to 15, characterized by the above.

17. The partial subject includes a head. The imaging device according to any one of claims 10 to 16, characterized by the above.

18. The partial subject includes a face. The imaging device according to any one of claims 10 to 17, characterized by the above.

19. When the state of the line-of-sight detection means is the first state, the control means changes the state of the line-of-sight detection means to the second state if a subject having a size smaller than the threshold is detected by the subject detection means even if the specific type of subject is not detected. The imaging device according to any one of claims 1 to 18, characterized by the above.

20. When the state of the line-of-sight detection means is the first state, the control means does not change the state of the line-of-sight detection means if a subject having a size smaller than the threshold is detected by the subject detection means without detecting the specific type of subject and if a plurality of subjects are detected. The imaging device according to claim 19, characterized by the above.

21. The imaging device further includes an eyepiece detection means for detecting an eyepiece to the viewfinder. The control means controls the state of the eyepiece detection means according to the state of the line-of-sight detection means. The imaging device according to any one of claims 1 to 20, characterized by the above.

22. The second state is a state in which the operation period of the line-of-sight detection means is longer than that of the first state. The control means When the state of the line-of-sight detection means is the first state, controls the state of the eyepiece detection means to a third state. When the state of the line-of-sight detection means is the second state, control the state of the eyepiece detection means to a fourth state in which the cycle of operation of the eyepiece detection means is shorter than the third state. The imaging device according to claim 21, characterized in that.

23. When the remaining battery level of the imaging device is more than the threshold value, the control means does not change the state of the line-of-sight detection means based on the detection result of the subject detection means, and when the remaining battery level is less than the threshold value, the control means changes the state of the line-of-sight detection means based on the detection result of the subject detection means. The imaging device according to any one of claims 1 to 22, characterized in that.

24. When the imaging device is not set to a specific mode, the control means does not change the state of the line-of-sight detection means based on the detection result of the subject detection means, and when the imaging device is set to the specific mode, the control means changes the state of the line-of-sight detection means based on the detection result of the subject detection means. The imaging device according to any one of claims 1 to 22, characterized in that.

25. Imaging means, Line-of-sight detection means for detecting the line of sight of the user, Subject detection means for detecting a subject from the image captured by the imaging means, When the state of the line-of-sight detection means is the first state for detecting the line of sight, Even if a plurality of human bodies and one or more balls are detected by the subject detection means, the state of the line-of-sight detection means is not changed. When one human body and one or more balls are detected by the subject detection means, change the state of the line-of-sight detection means to a second state in which the power consumption of the line-of-sight detection means is smaller than that in the first state. Control means An imaging device, characterized by comprising.

26. A control method for an imaging device having imaging means and line-of-sight detection means for detecting the line of sight of a user, A step of detecting a subject from the image captured by the imaging means, When the state of the line-of-sight detection means is the first state for detecting the line of sight, and a specific type of subject is detected from the image, a step of changing the state of the line-of-sight detection means to a second state in which the power consumption of the line-of-sight detection means is smaller than that in the first state. A control method, characterized by comprising.

27. A control method for an imaging device having imaging means and line-of-sight detection means for detecting the line of sight of a user, A step of detecting a subject from the image captured by the imaging means, When the state of the line-of-sight detection means is the first state for detecting the line of sight, even if a plurality of human bodies and one or more balls are detected from the image, the state of the line-of-sight detection means is not changed, when one human body and one or more balls are detected from the image, changing the state of the line-of-sight detection means to a second state in which the power consumption of the line-of-sight detection means is smaller than that in the first state; A control method characterized by comprising:

28. A program for causing a computer to function as each means of the imaging device according to any one of Claims 1 to 25.

29. A computer-readable storage medium storing a program for causing a computer to function as each means of the imaging device according to any one of Claims 1 to 25.

Images