Imaging device and method for control

Abstract

To provide an imaging device which can provide a function of supporting imaging of a moving object which makes a continuous detection difficult, and a method for controlling the imaging device.SOLUTION: The imaging device enlarges a focus detection region when an object region is not detected from a taken image or when the reliability that a detected object region corresponds to an intended object is determined to be low. The imaging device enlarges a focus detection region on the basis of the distance between a set focus detection region and a detected object region acquired in the past.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an imaging device and a method for controlling the same, and more particularly to autofocus detection (AF) technology. [Background technology]

[0002] A subject tracking function is known that achieves focus detection that follows the movement of a specific subject by continuously performing subject area detection and setting a focus detection area for the detected subject area (Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2009-284462 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The subject tracking function becomes difficult to maintain if the target subject area is no longer detected. For example, in swimming scenes, the subject tracking function targeting the swimmer's face may not work properly if the swimmer's head is in and out of the water or their face is covered by their hand.

[0005] If the subject tracking function is unavailable, you will need to adjust the camera's shooting range to follow the subject's movement so that the area you want to focus on (the swimmer's face in the example above) is within the focus detection area. This is similar to panning and is not easy.

[0006] In one embodiment, the present invention provides an imaging device and a control method thereof that can provide a function to support the shooting of moving subjects that are difficult to continuously detect. [Means for solving the problem]

[0007] The above objective is achieved by an imaging device comprising: detection means for detecting a subject area from a captured image; acquisition means for acquiring the distance between a set focus detection area and the detected subject area; and magnification means for expanding the focus detection area when the subject area is not detected or when it is determined that the reliability of the detected subject area corresponding to the intended subject is low, wherein the magnification means expands the focus detection area based on the distance previously acquired by the acquisition means. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging device and a control method therefor that can provide a function to support the shooting of moving subjects that are difficult to continuously detect. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram showing an example of the functional configuration of a digital camera as an example of an image processing apparatus according to the embodiment. [Figure 2] A schematic diagram showing the pixel arrangement in the embodiment. [Figure 3] schematic plan view and schematic cross-sectional view of a pixel in the embodiment [Figure 4] A diagram illustrating the correspondence between the pixels of the image sensor in the embodiment and the pupil intensity distribution. [Figure 5] A diagram showing the relationship between the amount of defocus and the amount of image displacement in the embodiment. [Figure 6] Diagram showing the focus detection region in the embodiment. [Figure 7] Flowchart relating to the operation of the shooting mode in the embodiment [Figure 8] Flowchart relating to imaging and recording processes in the embodiment [Figure 9] Flowchart relating to subject tracking AF processing in the embodiment [Figure 10] Flowchart relating to subject detection and tracking processing in the embodiment [Figure 11] Flowchart illustrating the process of setting the focus detection region in the first embodiment. [Figure 12] Figure for explaining the method of setting the focus detection area in the first embodiment [Figure 13] Figure for explaining the method of setting the focus detection area in the first embodiment [Figure 14] Figure for explaining the method of setting the focus detection area in the first embodiment [Figure 15] Figure for explaining the method of setting the focus detection area in the first embodiment [Figure 16] Table for explaining the setting of the focus detection area in the first embodiment [Figure 17] Flowchart for explaining the predictive AF processing in the first embodiment [Figure 18] Figure for explaining the image plane movement amount of the subject and the prediction curve in the first embodiment [Figure 19] Flowchart for explaining the setting process of the focus detection area in the second embodiment [Figure 20] Figure showing an example of framing in which the detected subject area M and the index I substantially match in the scene where shooting is to be performed in the second embodiment [Figure 21] Figure showing an example of framing in which the index I and the subject area M are misaligned in the scene where shooting is to be performed in the second embodiment [Figure 22] Figure for explaining a different example of the scene where shooting is to be performed in the second embodiment [Figure 23] Figure for explaining the method of setting the focus detection area in the second embodiment [Figure 24] Figure for explaining the method of setting the focus detection area in the second embodiment [Figure 25] Figure for explaining the method of setting the focus detection area in the second embodiment[[ID=四十二]] [Figure 26] Figure for explaining the method of setting the focus detection area in the second embodiment [Figure 27] Figure for explaining the method of setting the focus detection area in the second embodiment

Mode for Carrying Out the Invention

[0010] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0011] In the following embodiments, the present invention will be described in relation to cases where it is implemented using an imaging device such as a digital camera. However, an imaging function is not essential to the present invention, and it can be implemented with any electronic device. Such electronic devices include computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented with other electronic devices as well.

[0012] ●(Overall structure) Figure 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing apparatus according to this embodiment. The digital camera 100 (hereinafter simply referred to as camera 100) includes a first lens group 101, an aperture 102, a second lens group 103, and a third lens group 104, and has an imaging optical system that forms an optical image of a subject on the imaging surface of an image sensor 107.

[0013] The first lens group 101 is positioned furthest forward (on the subject side) of the multiple lens groups included in the imaging optical system and is movable along the optical axis OA. The position of the first lens group 101 is controlled by the zoom actuator 111. The zoom actuator 111 moves the first lens group 101 and the second lens group 103 in conjunction in the optical axis direction by, for example, driving a cam cylinder (not shown).

[0014] The aperture 102's opening amount can be adjusted by the aperture actuator 112, and it functions as a mechanical shutter when capturing still images. Alternatively, a separate mechanical shutter and a shutter actuator for driving the mechanical shutter may be provided.

[0015] The second lens group 103 moves along the optical axis OA, integrally with the aperture 102 and in conjunction with the first lens group 101. The angle of view (focal length) of the imaging optical system is determined by the positions of the first lens group 101 and the second lens group 103. The third lens group 105 is movable along the optical axis OA. The position of the third lens group 105 is controlled by the focus actuator 114. The position of the third lens group determines the focal distance of the imaging optical system. The third lens group is called the focus lens.

[0016] The optical low-pass filter 106 is provided to reduce false colors and moiré patterns that occur in the captured image. The image sensor 107 is a CMOS image sensor or CCD image sensor having a rectangular pixel array (also called a pixel area) in which, for example, m pixels in the horizontal direction and n pixels in the vertical direction are arranged in two dimensions. Each pixel is provided with, for example, a color filter according to the primary color Bayer array and an on-chip microlens. The image sensor 107 may also be a three-chip color image sensor.

[0017] In this embodiment, the imaging optical system, zoom actuator 111, aperture actuator 112, focus actuator 114, focus drive circuit 126, aperture drive circuit 128, and zoom drive circuit 129 are provided on the camera 100. However, these components may also be provided on a lens unit that is detachable from the camera 100.

[0018] The flash 115 is a light source that illuminates the subject. The flash 115 is equipped with a flash illumination device using a xenon tube or a continuously emitting LED (light-emitting diode). The AF (autofocus) auxiliary light source 116 projects a predetermined pattern image through a projection lens. This improves the focus detection capability for low-luminance or low-contrast subjects.

[0019] The CPU 121 controls the operation of the entire imaging device 100. The CPU 121 includes an A / D converter, a D / A converter, and a communication interface circuit. The CPU 121 controls various parts of the imaging device 100 by reading programs stored in the ROM 135 into the RAM 136 and executing them, thereby realizing the functions of the imaging device 100, such as autofocus detection (AF), imaging, image processing, and recording. In addition, some of the functions realized by the CPU 121 by executing programs may be implemented by hardware circuits separate from the CPU 121. Furthermore, reconfigurable circuits such as FPGAs may be used in some circuits. For example, in order to shorten the time required for calculations for focus detection, which will be described later, some of the calculations may be performed by dedicated hardware circuits.

[0020] The communication interface circuit provided by the CPU 121 supports one or more standards for wired and wireless communication. The imaging device 100 can communicate with external devices directly or via other devices through the communication interface circuit.

[0021] The flash control circuit 122 controls the illumination of the flash 115 in synchronization with the imaging operation. The auxiliary light source drive circuit 123 controls the illumination of the AF auxiliary light source 116 in synchronization with the focus detection process. The image sensor drive circuit 124 controls the imaging operation of the image sensor 107 and performs A / D conversion on the signals obtained from the imaging operation and transmits them to the CPU 121.

[0022] The image processing circuit 125 can apply various image processing operations to image data, such as gamma conversion, color interpolation, encoding, decoding, evaluation value generation, and feature region detection. Furthermore, the image processing circuit 125 acquires pose information for each subject region detected by the subject detection unit 140, which will be described later.

[0023] Whether the image processing circuit 125 should acquire posture information is predetermined depending on the type of subject. For example, if the subject is a human, the image processing circuit 125 will acquire the positions of multiple joints as posture information of the subject. The image processing circuit 125 will detect the positions of joints such as the top of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles. However, the image processing circuit 125 may detect only some of these joint positions, or it may detect other joint positions. In addition to joint positions, it may also detect information such as the axes connecting the joints and the position of the subject's center of gravity as posture information. The image processing circuit 125 can detect any information related to the subject's posture as posture information.

[0024] The image processing circuit 125 determines the main subject using the positional information of the detected feature region and the acquired pose information of the subject. The image processing circuit 125 may use the result of determining the main subject for other image processing (for example, white balance adjustment processing). The image processing unit 125 stores the image data in RAM 136, associating it with the pose information of the subject detected from the image data, the position and size of the feature region, the center of gravity of the subject determined to be the main subject, and positional information such as the face and pupils.

[0025] The focus drive circuit 126 drives the focus actuator 114 based on a command provided by the CPU 121, which includes the amount and direction of drive of the focus lens. As a result, the third lens group 105 moves along the optical axis OA, and the focusing distance of the imaging optical system changes.

[0026] The aperture drive circuit 128 drives the aperture actuator 112 to control the aperture diameter and opening / closing of the aperture 102. The zoom drive circuit 129 drives the zoom actuator 111, for example in response to user instructions, to move the first lens group 101 and the second lens group 103 along the optical axis OA, thereby changing the focal length (angle of view) of the imaging optical system.

[0027] The display unit 131 has, for example, an LCD (liquid crystal display device). The display unit 131 displays information related to the imaging mode of the imaging device 100, a preview image before imaging, a confirmation image after imaging, or an indicator of the focus detection area (AF frame), an image showing the focus status when focus is detected, etc.

[0028] The control unit 132 consists of a power switch, a shutter release switch, a zoom operation switch, and an imaging mode selection switch. The shutter release switch has two switches: SW1 which is ON when half-pressed and SW2 which is ON when fully pressed. The recording medium 133 is, for example, a semiconductor memory card that can be attached to and removed from the imaging device 100, and still image data and video data obtained by imaging are recorded on the recording medium 133.

[0029] If the display unit 131 is a touch display, a touch panel or a combination of the touch panel and a GUI displayed on the display unit 131 may be used as the operation unit 132. For example, if a tap operation on the touch panel is detected during live view display, the system can be configured to detect the focus using the image area corresponding to the tap position as the focus detection area.

[0030] Furthermore, the image processing circuit 125 can calculate contrast information from the captured image data, and the CPU 121 can perform contrast AF. In contrast AF, the focus lens group 105 is moved to sequentially calculate contrast information while changing the focus distance of the imaging optical system, and the focus lens position where the contrast information peaks is set as the focus position.

[0031] Thus, the imaging device 100 is capable of performing both image plane phase-detection AF and contrast AF, and can be selectively used or combined depending on the situation.

[0032] The subject detection unit 140 can be configured, for example, using a CNN (Convolutional Neural Network). By configuring the CNN using parameters (dictionary data) generated by machine learning, it detects the region of a specific subject present in the image represented by the image data. The subject detection unit 140 may also be implemented using dedicated hardware configured to perform calculations using the CNN at high speed, such as a GPU (Graphics Processing Unit) or an NPU (Neural Processing Unit).

[0033] Machine learning to generate dictionary data can be performed using any known method, such as supervised learning. Specifically, a CNN can be trained using a dataset in which input images are associated with whether or not the target subject is present, for each type of subject. The trained CNN or its parameters can be stored as dictionary data in the dictionary data storage unit 141. Note that CNN training may be performed on a device other than the camera 100. When using the trained CNN for subject detection processing of captured images, an image of the same size as the input image used during CNN training is extracted from the captured image and input to the CNN. By sequentially changing the extraction position and inputting it to the CNN, the region in which the target subject is present can be estimated.

[0034] Furthermore, after detecting object regions in an image, other methods may be used to detect the object region, such as determining what type of subject it is using feature quantities specific to each type of subject. The configuration and training method of the neural network can be changed depending on the detection method adopted.

[0035] The subject detection unit 140 can be implemented by any known method, as long as it can output the number, location, size, and confidence level of areas in the input image that are estimated to contain a predetermined type of subject.

[0036] The subject detection unit 140 can apply subject detection processing for multiple types of subjects to a single frame of image data by repeatedly applying subject detection processing while switching dictionary data. The CPU 121 can determine which dictionary data to use for subject detection processing from among the multiple dictionary data stored in the dictionary data storage unit 141, based on a pre-set priority for the types of subjects and the settings of the camera 100.

[0037] The types of subjects may include, but are not limited to, the human body, parts or organs of the human body (face, eyes, torso, etc.), or non-human subjects (animals, inanimate objects (tools, vehicles, buildings, etc.)). Separate dictionary data will be prepared for subjects with different characteristics.

[0038] Dictionary data for detecting the human body may be prepared separately for detecting the human body (outline) and for detecting the organs of the human body. Dictionary data for detecting the organs of the human body may be prepared separately for each type of organ.

[0039] The motion detection unit 142 is a sensor that detects, for example, the angular velocity around each of the x, y, and z axes, and the acceleration in the x, y, and z axis directions. The CPU 121 can detect the movement of the camera 100 based on the output of the motion detection unit 142. The detected movements include movements intended by the photographer, such as framing, and movements not intended by the photographer, such as camera shake. Here, framing refers to operations that move the shooting range in a constant direction without changing the angle of view of the camera 100, including panning and / or tilting. The CPU 121 uses the detected movement of the camera 100 for image stabilization and setting the focus detection area, which will be described later.

[0040] ●(Image sensor) Referring to FIGS. 2 and 3, the pixel array and pixel structure of the image sensor 107 will be described. In FIG. 2, the horizontal direction is the x direction (horizontal direction), the vertical direction is the y direction (vertical direction), and the direction orthogonal to the x direction and the y direction (direction perpendicular to the paper surface) is the z direction (optical axis direction). The example shown in FIG. 2 shows the pixel (unit pixel) array of the image sensor 107 in the range of 4 columns × 4 rows, and the sub-pixel array in the range of 8 columns × 4 rows.

[0041] The pixel group 200 of 2 columns × 2 rows has, for example, a pixel 200R having a spectral sensitivity of the first color R (red) at the upper left position, pixels 200G having a spectral sensitivity of the second color G (green) at the upper right and lower left positions, and a pixel 200B having a spectral sensitivity of the third color B (blue) at the lower right position. Further, each pixel (unit pixel) is composed of a plurality of sub-pixels including a first sub-pixel 201 and a second sub-pixel 202 (from the first sub-pixel to the N LF th sub-pixel) divided into 2 (division number N LF = Nx × Ny) in the x direction and 1 (Ny division) in the y direction.

[0042] In the example shown in FIG. 2, by dividing each pixel of the image sensor 107 into two sub-pixels arranged horizontally, from the image signal obtained by one imaging, the number of viewpoint images equal to the division number N LF and an imaging image obtained by synthesizing all the viewpoint images can be generated. Note that the pixels may be divided in two directions, and there is no limitation on the division number for each direction. Therefore, the viewpoint image can be said to be an image generated from the signals of some of the plurality of sub-pixels, and the imaging image can be said to be an image generated from the signals of all the sub-pixels. In this embodiment, as an example, the pixel periods P in the horizontal and vertical directions of the image sensor 107 are 6 μm, the horizontal pixel number N H = 6000, and the vertical pixel number N V = 4000. Therefore, the total pixel number N = N H × N V = 24 million. Also, if the horizontal period P S of the sub-pixels is 3 μm, the total number of sub-pixels N S = N H × (P / P S ) × N V = 48 million.

[0043] Figure 3(a) shows a plan view of one pixel 200G of the image sensor 107 shown in Figure 2, as seen from the light-receiving side (+z side) of the image sensor 107. The z-axis is set perpendicular to the plane of the paper in Figure 3(a), and the front side is defined as the positive direction of the z-axis. The y-axis is set vertically perpendicular to the z-axis, with the top being the positive direction of the y-axis, and the x-axis is set horizontally perpendicular to the z-axis and y-axis, with the right side being defined as the positive direction of the x-axis. Figure 3(b) shows a cross-sectional view as seen from the -y side along the aa cutting line in Figure 3(a).

[0044] As shown in Figures 3(a) and 3(b), each pixel 200G has a microlens 305 formed on the light-receiving surface side (+z direction), and the incident light is focused by this microlens 305. Furthermore, multiple photoelectric conversion units are formed, consisting of a first photoelectric conversion unit 301 and a second photoelectric conversion unit 302, each divided into two sections in the x (horizontal) direction and one section in the y (vertical) direction, resulting in a total of two divisions. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 correspond to the first sub-pixel 201 and the second sub-pixel 202 in Figure 2, respectively. More generally, the photoelectric conversion unit of each pixel is divided into Nx sections in the x direction and Ny sections in the y direction, with a total of N divisions for the photoelectric conversion unit. LF When = Nx × Ny, the 1st to Nth LF The photoelectric conversion unit is the 1st to Nth LF It corresponds to a sub-pixel.

[0045] The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are two independent pn-junction photodiodes, each composed of a p-type well layer 300 and two divided n-type layers 301 and 302. If necessary, an intrinsic layer may be sandwiched between them to form a PIN structure photodiode. A microlens 305 and a color filter 306 are formed between the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 in each pixel. If necessary, the spectral transmittance of the color filter 306 may be changed for each pixel or each photoelectric conversion unit, or the color filter may be omitted.

[0046] Light incident on pixel 200G is focused by microlens 305 and further spectrally separated by color filter 306 before being received by first photoelectric conversion unit 301 and second photoelectric conversion unit 302, respectively. In the first photoelectric conversion unit 301 and second photoelectric conversion unit 302, electron-hole pairs are generated according to the amount of light received, and after separation in the depletion layer, electrons are accumulated. Meanwhile, holes are discharged to the outside of the image sensor 107 through a p-type well layer connected to a constant voltage source (not shown). The electrons accumulated in the first photoelectric conversion unit 301 and second photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via a transfer gate and converted into a voltage signal.

[0047] In this embodiment, the microlens 305 corresponds to the optical system in the image sensor 107. The optical system in the image sensor 107 may be configured to use a microlens as in this embodiment, or it may be configured to use materials with different refractive indices, such as waveguides. Furthermore, the image sensor 107 may be a back-illuminated image sensor with circuits etc. on the side opposite to the side having the microlens 305, or it may be a stacked image sensor with some circuits such as the image sensor drive circuit 124 and the image processing circuit 125. In addition, materials other than silicon may be used as the semiconductor substrate, and for example, organic materials may be used as the photoelectric conversion material.

[0048] ●(pupil division) Next, the pupil division function of the image sensor 107 in this embodiment will be described with reference to Figures 4 and 5(a). Figure 4 shows a cross-sectional view of the aa cross-section of pixels 200G arranged on the image sensor 107 of this embodiment shown in Figure 3(a), viewed from the +y side, and the pupil plane located at a distance Z away from the imaging surface 600 of the image sensor 107 in the z-axis direction (direction of the optical axis OA). In Figure 4, the x and y axes of the cross-sectional view are inverted compared to Figure 3 in order to correspond with the coordinate axes of the exit pupil plane. The imaging surface 600 of the image sensor 107 is located at the imaging plane of the imaging optical system.

[0049] The pupil intensity distribution (first pupil intensity distribution) 501 is in a substantially conjugate relationship with the light-receiving surface of the first photoelectric conversion unit 301, whose centroid is eccentric in the -x direction, via the microlens 305. Therefore, the first pupil intensity distribution 501 corresponds to the pupil region that can be received by the first sub-pixel 201. The centroid of the first pupil intensity distribution 501 is eccentric on the pupil surface towards the +xp side. Similarly, the pupil intensity distribution (second pupil intensity distribution) 502 is in a substantially conjugate relationship with the light-receiving surface of the second photoelectric conversion unit 302, whose centroid is eccentric in the +x direction, via the microlens 305.

[0050] Therefore, the second pupil intensity distribution 502 corresponds to the pupil region that can receive light with the second sub-pixel 202. The centroid of the second pupil intensity distribution 502 is eccentric on the pupil plane towards -xp. Furthermore, the pupil intensity distribution 500 is the pupil region that can receive light with the entire pixel 200G when the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 (first sub-pixel 201 and second sub-pixel 202) are combined. In other words, the first pupil intensity distribution 501 is eccentric on the pupil plane towards +xp with respect to the center of the pupil intensity distribution 500, and the second pupil intensity distribution 502 is eccentric on the pupil plane towards -xp with respect to the center of the pupil intensity distribution 500.

[0051] Furthermore, in Figure 4, the pupil region 500 is the pupil region that can receive light with the entire pixel 200G, which includes the photoelectric conversion unit 301 and the photoelectric conversion unit 302 (first sub-pixel 201 and second sub-pixel 202).

[0052] Figure 5(a) shows a schematic diagram illustrating the correspondence between the image sensor and the pupil division in this embodiment. The light beam passing through the different pupil regions, the first pupil region 501 and the second pupil region 502, is incident on each (imaging) pixel of the image sensor from the imaging surface 800 at different angles and is received by the 2×1 divided photoelectric conversion units 301 and 302. In this embodiment, the pupil region is divided into two horizontally, but if necessary, the pupil may be divided vertically.

[0053] The image sensor 107 has an arrangement of imaging pixels, each having a first sub-pixel 201 that receives a light beam passing through the first pupil region 501 of the imaging optical system, and a second sub-pixel 202 that receives a light beam passing through the second pupil region 502 of the imaging optical system, which is different from the first pupil region. Therefore, the imaging pixels receive a light beam passing through the pupil region 500, which is the sum of the first pupil region 501 and the second pupil region 502 of the imaging optical system.

[0054] Furthermore, not all pixels of the image sensor 107 have multiple photoelectric conversion units (sub-pixels); instead, an imaging pixel, a pixel having only a first sub-pixel, and a pixel having only a second sub-pixel may be individually arranged. Alternatively, a pixel having two sub-pixels and a pixel having one photoelectric conversion unit equivalent to two sub-pixels may be arranged.

[0055] In this embodiment, the image processing circuit 125 generates a single phase-difference AF image data (first focus detection signal (image A)) by concatenating multiple signals obtained by adding the outputs of four first sub-pixels 201 for each repeating unit of the color filter shown in Figure 2. The image processing circuit 125 also generates a second focus detection signal (image B) by concatenating multiple signals obtained by adding the outputs of four second sub-pixels 202 for each repeating unit of the color filter.

[0056] Thus, the first and second sub-pixels 201 and 202 are used to generate the focus detection signal and are therefore also called focus detection pixels. By adding the outputs of the same type of sub-pixels present within the repeating unit of the color filter, a signal reflecting the R, G, and B color components is obtained, and it can be treated as a luminance signal with less bias in spectral sensitivity. When obtaining an image, the signal obtained is the sum of the output of the first sub-pixel 201 and the output of the second sub-pixel 202 on a pixel-by-pixel basis.

[0057] ●(Relationship between defocus amount and image displacement amount) This section describes the relationship between the amount of image shift of a pair of focus detection signals (image A and image B) generated by the image processing circuit 125 of this embodiment and the amount of defocus (degree of focus) of the optical system relative to the subject. Figure 5(b) shows a schematic relationship between the amount of defocus and the amount of image shift between the first focus detection signal (image A) and the second focus detection signal (image B). An image sensor 107 is arranged on the imaging surface 800, and as described with reference to Figures 4 and 5(a), the exit pupil of the imaging optical system is divided into a first pupil region 501 and a second pupil region 502.

[0058] The magnitude of the defocus amount d, |d|, is the distance from the image formation position of the subject to the image sensor 800. When the defocus amount d is negative (d<0), it means the image formation position of the subject is on the subject side of the image sensor 800, which is a front-focus state. When it is positive (d>0), it means the image formation position of the subject is on the opposite side of the image sensor 800, which is a back-focus state. When the image formation position of the subject is on the image sensor 800, which is a focused state, the magnitude of the defocus amount d is 0. In Figure 5(a), subject 801 is in focus (d=0) and subject 802 is shown as an example of a front-focus state (d<0). The front-focus state (d<0) and the back-focus state (d>0) together are called a defocus state (|d|>0).

[0059] In the front-focus state (d<0), the light beam from the subject 802 that has passed through the first pupil region 501 (second pupil region 502) is once focused at a position closer to the subject than the imaging plane 800. Then, it spreads out with a width Γ1 (Γ2) centered on the centroid position G1 (G2) of the light beam, resulting in a blurred image on the imaging plane 800. The blurred image is converted into an electrical signal by the first sub-pixel 201 (second sub-pixel 202) in each of the multiple pixels that receive the light. As described above, the phase-detection AF unit generates the first focus detection signal (image A) from the signal of the first sub-pixel 201 and the first focus detection signal (image B) from the signal of the second sub-pixel 202. Therefore, image A (image B) is recorded on the imaging plane 800 at the centroid position G1 (G2) as a subject image with the subject 802 blurred with a width Γ1 (Γ2).

[0060] The blur width Γ1 (Γ2) of the subject image increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|. Similarly, the magnitude of the image displacement amount p (= difference in the centroid position of the light beam G1-G2) of the subject image between the first focus detection signal and the second focus detection signal |p| also increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|. In the case of back focus (d>0), the relationship between the magnitude of the defocus amount |d|, the blur width of the subject image, and the image displacement amount p is the same, except that the direction of image displacement between image A and image B is opposite to that of the front focus state.

[0061] Therefore, as the amount of defocus increases, the amount of image shift between image A and image B also increases. The image sensor 107 that generates the first focus detection signal and the second focus detection signal, and the CPU 121 that calculates the amount of defocus from the amount of image shift p, function as focus detection means.

[0062] ●(Focus detection area) The pixel area of ​​the image sensor 107 used to generate the first focus detection signal and the second focus detection signal (focus detection area) will be described. Figure 6 shows an example of a unit focus detection area set in the effective pixel area 1000 of the image sensor 107, superimposed with the indicator (AF frame) of the focus detection area displayed on the display unit 131 when focus is detected. The indicator is displayed to inform the user of possible unit focus detection area candidates, the currently set unit focus detection area, the unit focus detection area selected by the camera 100, etc. The unit focus detection area on which the indicator is displayed does not necessarily coincide with the unit focus detection area used by the camera 100 for focusing control.

[0063] In this embodiment, a total of 25 unit focus detection regions are set, five in the row (x) direction and five in the column (y) direction. However, this is merely an example, and more or fewer unit focus detection regions may be set. Furthermore, the size, position, and spacing of the unit focus detection regions may vary.

[0064] In Figure 6, the nth horizontal and mth vertical unit focus detection region is represented as A(n,m), and the rectangular frame-shaped index representing the unit focus detection region A(n,m) is represented as I(n,m). The signals obtained from the first sub-pixel 201 and the second sub-pixel 202 within the unit focus detection region generate images A and B, which are used to detect the amount of defocus in that unit focus detection region. The index I(n,m) is usually superimposed on the live view image.

[0065] Here, the unit focus detection region A(n,m) where the index I(n,m) is not displayed is used as a supplement when the main subject moves out of the unit focus detection region A(n,m) where the index I(n,m) is displayed.

[0066] In this embodiment, the focus detection region 1001 can be set to include a unit focus detection region A(n,m) where the index I(n,m) is displayed and a unit focus detection region A(n,m) where the index I(n,m) is not displayed. In Figure 6, the focus detection region 1001 is set to include all 25 unit focus detection regions, but there may be unit focus detection regions that are not included in the focus detection region 1001.

[0067] In the case where every pixel in the effective pixel area 1000, such as in the image sensor 107, has a first sub-pixel 201 and a second sub-pixel 22, the position and size of the focus detection area may be dynamically set. For example, focus detection may be performed using an area that includes multiple unit focus detection areas centered on a position specified by the user. In this embodiment, the unit focus detection area is set so that a higher resolution focus detection result can be obtained when acquiring the defocus map described later. For example, each of the 9600 areas obtained by dividing the effective pixel area 1000 into 120 horizontally and 80 vertically is set as a unit focus detection area. The focus detection area can then be set to include a unit focus detection area A on which index I is displayed, and the surrounding unit focus detection area A on which index I is not displayed.

[0068] ●(Operation in imaging mode) The operation of camera 100 in imaging mode will be explained using the flowchart shown in Figure 7. When camera 100 is powered on and the startup process is completed, it operates in imaging mode.

[0069] In imaging mode, the camera 100 continuously captures video and displays the obtained video on the display unit 131, allowing the display unit 131 to function as an EVF while waiting for user input.

[0070] In S1, the CPU 121 starts driving the image sensor 107 via the image sensor drive circuit 124 in order to capture video for display on the display unit 131. Thereafter, the image sensor 107 outputs an analog image signal at a predetermined frame rate.

[0071] The CPU 121 acquires an analog image signal for one frame from the image sensor 107, and then applies correlated double sampling, A / D conversion, etc., to generate a digital image signal. The CPU 121 outputs the digital image signal to the image processing circuit 125. The image processing circuit 125 applies demosaicing and other processes to the digital image signal to generate image data for display. The image processing circuit 125 writes the image data for display to, for example, the video memory area of ​​RAM 136. The image processing circuit 125 also generates evaluation values ​​used for AE processing from the digital image signal and outputs them to the CPU 121. Furthermore, for each of the multiple unit focus detection areas, the image processing circuit 125 generates first and second focus detection signals based on the signals read from the pixels included in the unit focus detection area and outputs them to the CPU 121.

[0072] Furthermore, if the first and second sub-pixels 201 and 202 are configured as separate pixels (not sharing the same microlens), the pixel coordinates from which a signal is obtained from the first sub-pixel 201 will be different from the pixel coordinates from which a signal is obtained from the second sub-pixel 202. Therefore, the image processing circuit 125 interpolates the signals so that the signal pairs of the first and second sub-pixels 201 and 202 exist at the same pixel position to generate the first and second focus detection signals.

[0073] In S2, the CPU 121 supplies display image data stored in the video memory area of ​​the RAM 135 to the display unit 131, displaying it as a single frame of live view image. The user can adjust the imaging range, exposure conditions, etc., while viewing the live view image displayed on the display unit 131. The CPU 121 determines the exposure conditions based on evaluation values ​​obtained from the image processing circuit 125, and displays on the display unit 131 an image showing the determined exposure conditions (shutter speed, aperture value, imaging ISO sensitivity) and the image of indicator I shown in Figure 6, superimposed on the live view image. The photographer adjusts the shooting range so that the subject area to be in focus overlaps with indicator I in the live view image.

[0074] From this point onward, the CPU 121 executes operation S2 each time an image frame is captured. As a result, the display unit 131 functions as an EVF.

[0075] In S3, the CPU 121 determines whether a half-press operation (turning SW1 on) of the release switch included in the operation unit 132 has been detected. If the CPU 121 determines that SW1 is not turned on, it repeatedly executes S3. On the other hand, if it determines that SW1 is turned on, the CPU 121 executes S300.

[0076] In S300, CPU121 performs subject-tracking autofocus (AF) processing. In S300, CPU121 applies subject detection processing to the image data for display and determines the focus detection area. In addition, CPU121 performs predictive AF processing to suppress the decrease in AF accuracy caused by the time difference between executing AF processing and detecting the full press operation of the shutter release switch (SW2 on). Details of the operation in S300 will be described later.

[0077] In S5, CPU121 determines whether or not SW2 is turned on. If CPU121 determines that SW2 is not turned on, it executes S3. On the other hand, if it determines that SW2 is turned on, CPU121 executes the imaging and recording process in S400. Details of the operation in S400 will be described later.

[0078] In S7, the CPU 121 determines whether or not it has detected that the main switch included in the control unit 132 is off. If the CPU 121 determines that it has detected that the main switch is off, it terminates the operation of the imaging mode. On the other hand, if the CPU 121 determines that it has not detected that the main switch is off, it executes S3.

[0079] Although subject detection and AF processing were explained as being performed in response to the detection of SW1 being on, they can also be performed at a different time. If the subject tracking AF processing of the S300 is performed before the on status of SW1 is detected, it becomes possible to skip the half-press operation and start imaging immediately with a full-press operation.

[0080] ● (Imaging and recording processing) Next, using the flowchart shown in Figure 8, we will explain the imaging and recording process performed by the CPU 121 in S400 of Figure 7. In S401, the CPU 121 determines the exposure conditions (shutter speed, aperture value, imaging ISO sensitivity, etc.) through AE processing based on evaluation values ​​generated by the image processing circuit 125. The CPU 121 then controls the operation of each part to capture a still image according to the determined exposure conditions.

[0081] Specifically, the CPU 121 transmits the aperture value and shutter speed to the aperture drive circuit 128 to drive the aperture 102. The CPU 121 also controls the charge accumulation operation of the image sensor 107 through the image sensor drive circuit 124.

[0082] In step S402, the CPU 121 reads out one frame of analog image signals from the image sensor 107 via the image sensor drive circuit 124. Additionally, for pixels within the focus detection area, the signal from either the first or second sub-pixel 201 or 202 is also read out.

[0083] In S403, the CPU 121 performs A / D conversion on the signal read in S402 to obtain a digital image signal. The CPU 121 also applies defective pixel correction processing to the digital image signal using the image processing circuit 125. Defective pixel correction processing is a process that compensates for signals read from pixels from which normal signals cannot be read (defective pixels) by using signals read from surrounding normal pixels.

[0084] In step S404, the CPU 121 instructs the image processing circuit 125 to generate a still image data file for recording and first and second focus detection signals. The image processing circuit 125 applies image processing and encoding to the digital image signal after defective pixel correction processing to generate still image data for recording. Image processing may include, for example, demosaicing (color interpolation), white balance adjustment, gamma correction (gradation correction), color conversion, and edge enhancement. The image processing circuit 125 also applies encoding processing to the still image data in a manner appropriate to the format of the data file that stores the still image data.

[0085] In S405, the CPU 121 records an image data file containing the still image data generated in S404 and the signals of the sub-pixels read from the focus detection area in S402 onto the recording medium 133.

[0086] In S406, the CPU 121 records the device characteristic information, which is characteristic information of the camera 100, on the recording medium 133, associating it with the image data file recorded in S405.

[0087] Device characteristic information includes, for example, the following information: • Imaging conditions (aperture value, shutter speed, ISO sensitivity, etc.) • Information regarding the image processing applied to the digital image signal by the image processing circuit 125. • Information regarding the light-receiving sensitivity distribution of the imaging pixels and sub-pixels of the image sensor 107. • Information regarding vignetting of the imaging light beam within Camera 100 • Information on the distance from the mounting surface of the imaging optical system in camera 100 to the image sensor 107. • Information regarding manufacturing tolerances for Camera 100

[0088] Information regarding the light-receiving sensitivity distribution of imaging pixels and sub-pixels (hereinafter simply referred to as light-receiving sensitivity distribution information) is information regarding the light-receiving sensitivity of the image sensor 107 according to the distance from the intersection point of the image sensor 107 and the optical axis. Since the light-receiving sensitivity depends on the microlenses 305 and photoelectric conversion units 301, 302 of the pixel, the information may also be information regarding these. Furthermore, the light-receiving sensitivity distribution information may also be information regarding the change in sensitivity with respect to the angle of incidence of light.

[0089] In S407, the CPU 121 records lens characteristic information, which is characteristic information of the imaging optical system, onto the recording medium 133, corresponding to the still image data file recorded in S405.

[0090] Lens characteristic information includes, for example, the following: • Information regarding the exit pupil • Information regarding the frame of the telescope tube, etc., that emits the light beam. • Information on focal length and F-number at the time of imaging • Information regarding aberrations in the imaging optical system • Information regarding manufacturing tolerances of imaging optics. • Position of the focus lens 105 during image capture (distance from subject)

[0091] Next, in S408, the CPU 121 records image-related information, which is information about the still image data, on the recording medium 133, corresponding to the still image data file recorded in S405. The image-related information includes, for example, information about the focus detection operation before imaging, information about the movement of the subject, and information about the focus detection accuracy.

[0092] In addition, in S406-S408, the CPU 121 may also save device characteristic information, lens characteristic information, and image-related information in the ROM 136, associating them with the image data file recorded in S405.

[0093] In step S409, the CPU 121 generates display image data by scaling the still image data in the image processing circuit 125 and displays it on the display unit 131. This allows the user to view the captured image. After a predetermined display time has elapsed, the CPU 121 terminates the imaging and recording process.

[0094] ●(Subject tracking AF processing) Next, the subject-tracking AF process in the S300 shown in Figure 7 will be explained using the flowchart shown in Figure 9.

[0095] In S301, the CPU 121 calculates the amount of image shift (phase difference) between the first and second focus detection signals generated in S2 for each of the multiple unit focus detection regions. The amount of image shift between signals can be determined as the relative position where the correlation between the signals is maximized. From the calculated amount of image shift, the CPU 121 calculates the amount of defocus, which is the degree of focus for each unit focus detection region.

[0096] As described above, in this embodiment, the effective pixel area 1000 is divided into 120 horizontally and 80 vertically, resulting in a total of 9600 areas, each of which is set as a unit focus detection area. The CPU 121 generates data (defocus map) that associates the defocus amount calculated for each unit focus detection area with the position of the area. The CPU 121 stores the generated defocus map, for example, in RAM 136.

[0097] In S302, the CPU 121 performs subject detection processing using the subject detection unit 140. The subject detection unit 140 detects areas of one or more types of subjects and outputs a detection result to the CPU 121 for each detected area, including the type of subject, the location and size of the area, and the reliability of the detection.

[0098] Furthermore, the CPU 121 performs a process to detect the position of the subject in the current frame (tracking process) based on the results of the subject detection process in the current frame and the results of the subject detection process in past frames. If the subject cannot be detected by the subject detection process using the pre-trained CNN in the subject detection unit 140, the CPU 121 can estimate the position of the subject in the current frame by tracking using other methods such as template matching. Details will be described later.

[0099] In S303, the CPU 121 sets the focus detection area based on the results of the subject detection process in S302. In this embodiment, the CPU 121 determines the main subject intended by the photographer from the information of the detected subject and the movement of the camera 100. The CPU 121 then sets a focus detection area that includes one or more appropriate unit focus detection areas in the area of ​​the determined main subject, and maintains a state of being in focus on the area of ​​the main subject. Details regarding the setting of the focus detection area will be described later.

[0100] In S304, CPU121 obtains the defocus amount of the unit focus detection region included in the focus detection region set in S303. The defocus amount obtained here may be the one calculated in S301, or it may be the defocus amount calculated again for the new frame. In addition, the defocus amounts of one or more unit focus detection regions located around the focus detection region set in S303 may also be obtained.

[0101] In S305, CPU121 performs predictive AF processing using the defocus amount obtained in S304 and previously obtained defocus amounts. This process is necessary when there is a time lag between the timing of focus detection and the timing of image exposure. It predicts the position of the subject a predetermined time after the timing of focus detection and performs AF control. Details will be described later.

[0102] In S306, the CPU 121 performs focus adjustment processing for the next frame image based on the amount of defocus predicted in S305 for the main subject. The CPU 121 converts the amount of defocus into the drive direction and drive amount of the focus lens, and controls the focus actuator 114 through the focus drive circuit 126 to drive the focus lens 105. Details will be described later. Once the driving of the focus lens 105 is complete, the CPU 121 terminates the subject tracking AF processing.

[0103] ● (Subject detection and tracking processing) Next, we will explain the details of the subject detection and tracking process in S302 of Figure 9 using the flowchart shown in Figure 10.

[0104] In S2000, the CPU 121 sets the dictionary data to be used by the subject detection unit 140 by determining the type of subject to be detected by the subject detection unit 140. The type of subject to be detected can be determined based on predetermined priorities, camera settings (e.g., shooting mode), etc. For example, the dictionary data storage unit 141 may store dictionary data for "people," "vehicles," and "animals." The types of subjects may be classified more finely. For example, instead of "animals," dictionary data for "dogs," "cats," "birds," and "cows" may be stored, or instead of "vehicles," dictionary data for "four-wheeled vehicles," "two-wheeled vehicles," "railways," and "airplanes" may be stored.

[0105] The CPU 121 sets dictionary data for a specific type of subject if the camera 100 is set to a shooting mode intended for that type of subject. For example, if portrait mode or sports mode is set, the CPU 121 sets the dictionary data for "people". Also, if "panning mode" is set, the CPU 121 sets the dictionary data for "vehicles".

[0106] If the camera 100 does not have a shooting mode set for a specific type of subject, the CPU 121 sets the subject dictionary data according to a predetermined priority. For example, it can set dictionary data for "people" and "animals".

[0107] The type of dictionary data and the method for determining which dictionary data to set are not limited to those described here. You may set one dictionary data or multiple dictionary data. If you use one dictionary data, you can detect subjects that are detectable by that single dictionary data with high frequency. If you set multiple dictionary data, you can detect multiple types of subjects by switching dictionaries for each frame. Furthermore, if processing time allows, you may detect multiple types of subjects in the same frame.

[0108] In S2001, the CPU 121 uses the subject detection unit 140 to apply subject detection processing to the image of the current frame. Here, the subject to be detected is assumed to be a "person". The subject detection unit 140 uses the "person" dictionary data stored in the dictionary data storage unit 141 to apply subject detection processing to the image of the current frame. The subject detection unit 140 outputs the detection result to the CPU 121. At this time, the CPU 121 may also display the subject detection result on the display unit 131. The CPU 121 also saves the detected subject area to the RAM 136.

[0109] When the "person" dictionary data is set, the subject detection unit 140 detects multiple types of regions related to a person with different granularities, such as the "whole body" region, the "face" region, and the "eyes" region. While it is desirable to detect local areas such as a person's eyes and face for use in focus detection and exposure control, there is a possibility that they cannot be detected if the face is not facing forward or is hidden by other subjects. On the other hand, it is unlikely that the whole body will not be detected at all. Therefore, by detecting multiple types of regions with different granularities, the probability of detecting some region of a "person" is increased.

[0110] Furthermore, the dictionary data can be configured to detect multiple types of regions with different levels of granularity for subjects other than people. For example, if the "vehicle" dictionary data is set, the subject detection unit 140 can configure the dictionary data to detect the region of the entire vehicle (including the occupants in the case of motorcycles and racing cars), as well as the region of the occupants' "heads" and / or "faces".

[0111] In S2002, CPU121 performs subject tracking by applying template matching to the current frame using the subject region most recently detected in S2001 as a template. The image of the subject region itself may be used as the template, or information obtained from the subject region, such as brightness information, color histogram information, and feature point information such as corners and edges, may be used as the template. Any known method can be used for matching and updating the template. As a result of the tracking process, CPU121 stores in RAM136 the position and size of the region (subject region) that is most similar to the template in the current frame, the movement of the region between frames, and the detection confidence level of the region.

[0112] The tracking process in S2002 may be executed only if no subject was detected in S2001. By detecting a subject area similar to a previously detected subject area in the current frame, stable subject detection and tracking can be achieved. The CPU 121 terminates the subject detection and tracking process upon completion of the tracking process.

[0113] ● (Setting the focus detection area) Next, we will explain the details of setting the focus detection area in S303 in Figure 9 using the flowchart shown in Figure 11. In S4000, CPU 121 obtains the results of the subject detection and tracking process (subject detection information) performed in S302 of Figure 9 from RAM 136. The subject detection information may include, for example, the position of the subject area (e.g., centroid coordinates), size, movement, and detection confidence level.

[0114] In S4001, the CPU 121 acquires information regarding the movement of the camera 100 from the output of the motion detection unit 142.

[0115] In S4002, CPU121 determines the reliability that the subject area for which information was acquired in S4000 is the subject intended by the photographer. CPU121 acquires the image coordinates of the unit focus detection area (or the indicator (AF frame) corresponding to the unit focus detection area) set by the user. The following describes the case where the image coordinates of the indicator are acquired, but the same procedure can be followed when acquiring the image coordinates of the unit focus detection area.

[0116] The CPU 121 calculates the distance on the image sensor 107 between the indicator and the subject area as an indicator of reliability that the subject intended by the photographer has been detected (that the detected subject area is the area of ​​the subject intended by the photographer). If multiple subject areas are detected, the distance can be calculated for the subject area with the highest detection reliability, or for the subject area closest to the indicator among subject areas with a detection reliability of a predetermined value or higher. If there are multiple indicators displayed, the CPU 121 calculates the distance between the subject area and the indicator closest to the subject area. In addition, if the position of the indicator is within the subject area, the distance may be set to 0.

[0117] The CPU 121 then determines that if the calculated distance is less than the threshold, it has a high degree of confidence that the subject intended by the photographer has been detected, and executes S4003. The threshold can be set, for example, as the distance between adjacent unit focus detection areas in both the x and y directions. If both the x and y components of the distance are greater than or equal to the threshold, it can be determined that the distance is greater than or equal to the threshold. However, this is just one example, and thresholds and distances can be determined using other methods.

[0118] On the other hand, if the calculated distance is above a threshold, the CPU 121 determines that the reliability of detecting the subject intended by the photographer is low and executes S4005. Furthermore, even if no subject area is detected, or if there are no subject areas with a detection reliability above the threshold, the CPU 121 can determine that the reliability of detecting the subject intended by the photographer is low.

[0119] The CPU 121 adds the judgment result from S4002 to the subject area information stored in RAM 136.

[0120] In S4003, CPU121 stores the distance (positional shift) between the subject area and the indicator, calculated in S4002, in RAM136. CPU121 stores the distance between the subject area and the indicator for a predetermined number of recent times in a time series. Since this time series data of distance depends on the movement of the subject and the photographer's skill, it is reset (deleted) and saving starts again when the shooting date or the type of subject changes.

[0121] In S4004, CPU121 sets the focus detection area if the subject intended by the photographer has been detected. In this case, CPU121 sets the focus detection area to include the unit focus detection area displaying the indicator and the unit focus detection area that exists within the subject area. It may also include unit focus detection areas adjacent to the subject area. After that, CPU121 executes S4007.

[0122] Meanwhile, in S4005, CPU121 acquires the history (time-series data) of the positional difference between the index calculated in S4003 and the subject area, which is stored in RAM136.

[0123] In S4006, CPU121 uses the time-series distance data acquired in S4005 to set the focus detection area when the subject intended by the photographer is detected. The time-series distance data reflects the influence of (1) the photographer's framing skills, (2) the subject's movement characteristics (movement speed, acceleration, etc.), and (3) the focal length of the imaging optical system corresponding to the amount of angle of view change during framing. For example, if the photographer's framing skills are low, the variation in distance tends to be greater than when the skills are high, and the average distance also tends to be higher.

[0124] The CPU 121 separates the individual distances contained in the time series data into x-direction and y-direction components and calculates the standard deviation for each direction. Then, the CPU 121 sets a focus detection region that includes the index position and has a length that is a predetermined multiple of the standard deviation in the x and y directions. For example, if the distances follow a normal distribution, by setting a focus detection region centered on the index position and having a size of twice the standard deviation for each direction, the subject can be placed within the focus detection region with a probability of approximately 95%.

[0125] The focus detection area needs to be set to include the subject the photographer intends to photograph. However, if the focus detection area is larger than the subject area, there is a possibility that the camera will focus on other subjects in the background or foreground of the intended subject. Therefore, the size of the focus detection area needs to be set appropriately. By dynamically determining the size of the focus detection area based on time-series distance data, it is possible to increase the likelihood of capturing the subject within the focus detection area while suppressing the possibility of focusing on other subjects.

[0126] The size of the focus detection region can also be determined dynamically by other methods. For example, if the number of distance samples in the time series data is small (below a threshold), the size of the focus detection region may be determined based on the standard deviation calculated without separating the distance into its directional component.

[0127] Furthermore, the focus detection area does not have to be rectangular, as long as it encompasses multiple unit focus detection areas. Therefore, the focus detection area can be set in any shape, such as a rhombus or an ellipse. It may also be an asymmetrical shape that grows larger in the direction of the subject's movement.

[0128] Furthermore, the focus detection area may be set considering the camera motion information acquired in S4001. For example, if the framing speed is a second speed which is faster than the first speed, the size of the focus detection area in the framing direction from the index position can be made larger than in the case of the first speed.

[0129] When the focus detection area is set in S4006, CPU121 executes S4007.

[0130] In S4007, CPU 121 selects a unit focus detection area A(n,m) included in the focus detection area set in S4004 or S4006. Then, CPU 121 displays an index I(n,m) corresponding to the selected unit focus detection area A(n,m), in addition to the already displayed index. This allows the photographer to know the focus detection area set by camera 100. Note that the user may be able to configure whether or not to display the additional index. Once processing S4007 is complete, CPU121 finishes setting the focus detection area.

[0131] ●(Specific example of setting the focus detection area) Next, using Figures 12 and 13, we will explain specific examples of setting the focus detection area in S4004 or S4006 in Figure 11. Figure 12 schematically illustrates a scene in which a person swimming butterfly moves horizontally from left to right, with the user setting a single unit focus detection area where indicator I is displayed, and framing the camera 100 so that the subject is positioned within indicator I. The detected subject area is indicated by M, the focus detection area by 1001, and the indicator by I.

[0132] At the point shown in Figure 12(a), indicator I is positioned slightly ahead of the subject's head in the direction of movement. In this scene, the subject's head is either hidden underwater or obscured by splashes or hands even if it is above water. Therefore, framing the shot so that indicator I overlaps with the head is not easy, and indicator I is positioned offset from the head.

[0133] In Figure 12(a), the subject's head is detected as the subject area M, and its distance to the index I is also close (below the threshold). Therefore, in S4002 of Figure 11, as explained earlier, the CPU 121 determines that there is a high degree of confidence that the subject area M corresponds to the intended subject. As a result, in S4004, the CPU 121 sets the focus detection area 1001 based on the subject area M. Here, an example is shown in which the focus detection area 1001 is set to be slightly larger than the subject area M, but the subject area M can also be set as the focus detection area 1001 as is.

[0134] Figure 12(b) shows a case where the head is above the water surface, but the subject area M is not detected due to the arms and splashes of water. In this case, at S4002 in Figure 11, the CPU 121 determines that the reliability of the subject area M corresponding to the intended subject is low. As a result, at S4006, the CPU 121 sets the focus detection area based on the time-series data of the distance between the previously detected subject area M and index I.

[0135] Figure 12(b) shows an example of a focus detection region 1001 set when the variability of the time-series data of the distance between the index and the subject area is not large. The distance between the index and the subject area has the horizontal direction, which is the direction of movement of the subject, as its main component. Therefore, the CPU 121 sets the focus detection region 1001 to be expanded around the index I, with the horizontal direction, which is the direction of movement of the subject, as its size. Although not shown in Figure 12(b), the CPU 121 sets the focus detection region 1001 so that the number of unit focus detection regions included in the horizontal direction is greater than in the case of Figure 12(a).

[0136] Figure 12(c) shows the case where the head moves below the water surface and the subject area M is not detected. In this case as well, the CPU 121 sets the focus detection area 1001 in the same way as shown in Figure 12(b). Here, we show an example of the focus detection area 1001 that is set when there is a large variation in the time series data of the distance between the indicator and the subject area.

[0137] For example, if the photographer's framing skills are poor, the subject will appear and disappear from view, resulting in greater variation in distance. This variation in distance is particularly large in the direction of the subject's movement (horizontal direction). By determining the size of the focus detection area according to the magnitude of the distance variation, a larger focus detection area is set for photographers with poor framing skills than the focus detection area set for photographers with good framing skills. In Figure 12(c), a larger focus detection area 1001 is set in both the horizontal and vertical directions than in Figure 12(b), but the size is particularly larger in the direction of the subject's movement (horizontal direction).

[0138] Figure 13 schematically illustrates a scene in which a person performing a diving competition moves vertically from top to bottom, with the user setting a single unit focus detection area where indicator I is displayed, and framing the camera 100 so that the subject is positioned within indicator I.

[0139] Figure 13(a) shows an example of the focus detection region 1001 set when the subject area M is detected. The subject area M is slightly offset from the index I, but the distance to the index I is close (below the threshold). Therefore, in S4002 of Figure 11, as explained earlier, the CPU 121 determines that there is a high reliability that the subject area M is the intended subject. As a result, in S4004, the CPU 121 sets the focus detection region 1001 based on the subject area M. Here, an example is shown in which the focus detection region 1001 is set to be slightly larger than the subject area M, but the subject area M may also be set as the focus detection region 1001 as is.

[0140] Figure 13(b) shows an example of the focus detection region 1001 that is set when the subject area M cannot be detected due to overlap between the arm and head, although the subject's posture is almost the same as in Figure 13(a). In this case, at S4002 in Figure 11, the CPU 121 determines that there is little confidence that the subject area M is the intended subject. As a result, at S4006, the CPU 121 sets the focus detection region based on time-series data of the distance between the previously detected subject area M and index I.

[0141] In the scene shown in Figure 13, the subject is moving vertically, so the distance between the indicator and the subject area is primarily vertical. Therefore, the CPU 121 sets the focus detection area 1001 to be enlarged around indicator I, with the vertical dimension being the direction of the subject's movement. Here, it is enlarged by the same amount vertically relative to indicator I, but it may be enlarged more depending on the direction of the subject's movement.

[0142] For example, Figure 14(a) shows another example of a focus detection area to be set when the subject area cannot be detected or when the distance between the indicator and the subject area exceeds a threshold. Here, since the direction of movement of the subject is to the right on the paper, when expanding the focus detection area horizontally, the amount or ratio of expansion is greater to the right of the indicator I than to the left.

[0143] Furthermore, the time-series data of the referenced distance can be used to determine whether the framing motion is lagging behind the subject's movement (for example, the average distance is negative) or ahead of it (for example, the average distance is positive). If the framing motion is determined to be lagging behind the subject's movement, the magnification or magnification ratio in the forward direction of movement may be increased compared to when it is determined to be ahead.

[0144] Figure 14(b) shows the same situation as Figure 14(a), but with index I overlapping with the subject's head, indicating that the framing is successful. In this case, the time-series data of distance indicates that the distance is small and the variation is small. In such cases, instead of unnecessarily expanding the focus detection area, a focus detection area 1001 can be set that is slightly expanded in the direction of the subject's movement.

[0145] Figures 15(a) and 15(b) show examples of focus detection regions 1001 set using essentially the same method as in Figures 14(a) and 14(b), except that the subject's direction of movement is vertical (from top to bottom). However, in the example in Figure 15, the subject's movement speed is greater than in the example in Figure 14, so the downward expansion in the direction of movement is greater than in the example shown in Figure 14.

[0146] The subject's movement speed can be detected using the shooting interval, the distance traveled between captured images, the subject distance, and the camera 100's movement information, but it may also be detected by other methods.

[0147] The time-series data of distance calculated in S4003 and stored in RAM136 may be used without resetting, for example, while it is determined that the same scene is being continuously photographed. The CPU121 can determine that the same scene is being continuously photographed, for example, based on the motion information of the camera 100, if it is determined that the camera 100 is repeatedly performing the same framing. Alternatively, the time-series data of distance may be reset when one framing is completed.

[0148] Furthermore, when setting the focus detection area 1001, information obtained from a neural network trained using dictionary data 141 and data from the motion detection unit 142 may be taken into consideration. This makes it possible to set a focus detection area 1001 that is better suited to the characteristics of each individual photographer.

[0149] ● (Setting the focus detection area according to distance and framing amount) Up to this point, we have described a method for determining the size of the focus detection area based on the variability (e.g., standard deviation) and magnitude (e.g., mean value) of the index position, subject area, and distance calculated multiple times over time. However, the size of the focus detection area can also be determined by other methods.

[0150] Figure 16 shows an example of determining the size (range) of the focus detection area by combining the magnitude of the distance between the indicator and the subject area (e.g., the average value) and the magnitude of the framing amount (e.g., the average value), which are obtained from time-series data. The magnitude of the framing amount (velocity or acceleration) can be determined from the movement of the camera 100 detected by the motion detection unit 142.

[0151] Here, the magnitude of distance and the magnitude of framing are each classified into two categories using predetermined thresholds, and the size of the focus detection area is determined according to the four possible combinations. Note that at least one of the two can be classified into three or more categories.

[0152] For example, the CPU 121 calculates a representative value (e.g., the average value) of the magnitude of the distance between the indicator and the subject area from the time-series data acquired in S4005. The CPU 121 also calculates the framing amount of the camera 100 based on the movement of the camera 100 detected by the motion detection unit 142. The CPU 121 classifies the representative value of the magnitude of the distance and the framing amount as "large" or "small" by comparing them with their respective thresholds. Based on the combination of classification results, the CPU 121 obtains one of the ranges A to D of the focus detection area by referring to a table stored in the ROM 135. Ranges A to D may be, for example, magnification ratios relative to a reference size (e.g., the size of indicator I).

[0153] Generally, when the distance between the indicator and the subject area is large, the magnification should be higher than when it is small. This is because a large distance suggests that the photographer's framing skills are low. Similarly, when the amount of framing is large, the magnification should be higher than when it is small. This is because a large amount of framing suggests that the subject is moving quickly. Therefore, among A to D, A is the largest and D is the smallest. The relative sizes of B and C may change depending on the threshold used for classification.

[0154] Note that the table in Figure 16 may be prepared separately for the x and y directions. In this case, the representative value of the distance and the magnitude of the framing amount should also be determined for both the x and y directions. By preparing magnifications for each direction, the focus detection area can be set with a magnification corresponding to the direction of movement of the subject.

[0155] ● (Predictive AF processing) Next, the details of the predictive AF processing in S305 in Figure 9 will be explained using the flowchart shown in Figure 17.

[0156] In S6001, the CPU 121 determines whether the subject is a moving object in the direction of the optical axis. If it is determined that the subject is a moving object in the direction of the optical axis, it executes S6002; otherwise, it executes S6009. Whether the subject is a moving object in the direction of the optical axis can be determined by referring to the time-series data of the defocus map and based on the change in the amount of defocus in the subject area or the change in the focal distance.

[0157] In S6002, CPU121 determines whether the image plane velocity of the subject is high or low. If it is determined to be high, it executes S6003; otherwise, it executes S6009. For example, CPU121 can determine that the image plane velocity is high if the inter-frame movement of the subject area (the difference in the amount of defocus between frames due to movement in the optical axis direction) is greater than or equal to a threshold.

[0158] In S6003, CPU121 sets the number of time-series data points used to predict the subject's position. Figure 18(a) shows an example of the change in the image plane position of the subject over time. The horizontal axis represents time, and the vertical axis represents the amount of image plane displacement. The black circles also show the time series of the amount of movement of the subject area detected in the past (defocus amount due to movement in the optical axis direction). Figure 18(a) shows the case where the amount of image plane displacement of the subject increases over time. The future position of the subject area is predicted from the time-series data.

[0159] When the image plane velocity or acceleration of the subject is low, increasing the number of time-series data points used for prediction does not increase the prediction error. Rather, increasing the number of data points can reduce the impact of the defocus error on the predicted position.

[0160] On the other hand, if the image plane velocity or acceleration of the subject is high, increasing the number of time-series data used for prediction will result in a larger use of data from periods when the image plane velocity is low, thus increasing the error in the predicted position.

[0161] Therefore, in this embodiment, when it is determined that the image plane velocity or acceleration is high, the number of time-series data points of the image plane movement of the subject used to calculate the prediction curve is reduced compared to when it is not high. By reducing the number of time-series data points, the number of older data points from periods when the image plane velocity or acceleration is low is reduced, which helps to suppress the decrease in the accuracy of the predicted position, especially for subjects whose velocity increases over time.

[0162] In S6004, CPU121 sets the focus movement range. The focus movement range is explained using Figure 18(b). In Figure 18(b), the horizontal axis is time, the vertical axis is image plane position, the solid line is the focus position, and the dotted line is the focus movement range.

[0163] When the subject is moving, it is easy for the subject to move out of the set focus detection area. If focusing control is performed while the subject is out of the focus detection area, the focus distance can change significantly. Therefore, when the subject returns to the focus detection area, the time required to focus on the subject increases. For this reason, the CPU 121 estimates the distance range in which the subject moves based on the subject's image plane movement speed and subject distance, and limits the range in which focus movement is possible (the acceptable range of change in focus distance).

[0164] Furthermore, the CPU 121 does not drive the focus lens if the predicted image plane position (distance) of the subject has changed beyond the acceptable range from the current focus distance. This prevents the subject from suddenly becoming blurred even if it unintentionally moves out of the focus detection area.

[0165] In S6005, CPU121 calculates the predicted position (distance) of the subject. For example, CPU121 can obtain the equation of a prediction curve using multivariate analysis (e.g., least squares method) with time-series data of past image plane position and time, and by substituting the time at which the still image is taken into the equation of the prediction curve, it can predict the image plane position of the subject.

[0166] In S6006, CPU121 changes the focus movement speed. CPU121 sets the focus's image plane movement speed according to the subject's image plane movement speed estimated from the subject's image plane position predicted in S6005 and time-series data of past image plane positions. In the example in Figure 18(a), since the subject's image plane movement speed is estimated to be increasing over time, CPU121 increases the focus's image plane movement speed.

[0167] In S6007, CPU121 determines whether the image plane position (distance) of the subject predicted in S6005 is within the focus movement range set in S6004. If CPU121 determines that the predicted image plane position of the subject is within the focus movement range, it executes S6008; otherwise, it terminates the predictive AF process without driving the focus lens.

[0168] In S6008, CPU121 drives the focus lens 105 to focus on the position predicted by S6005, and then terminates the predictive AF processing.

[0169] Furthermore, if the predicted image plane position of the subject is not determined to be within the focus movement range for a predetermined number of consecutive times (or for a predetermined period of time), S6008 may be executed. In this case, the CPU 121 resets the time-series data related to the image plane position of the subject and generates new time-series data.

[0170] As described above, according to this embodiment, if the subject area cannot be detected, the focus detection area is enlarged based on past data regarding the distance between the subject area and the focus detection area. Therefore, for example, when photographing a moving subject while framing the camera, it is possible to set a focus detection area of ​​a size appropriate to the photographer's framing skills. This reduces the risk of focusing on an unintended subject by unnecessarily enlarging the focus detection area, while increasing the likelihood that focus control will be performed on the intended subject even if the photographer's framing skills are low.

[0171] ●(Second Embodiment) Next, a second embodiment will be described. This embodiment is the same as the first embodiment, except for the operation of setting the focus detection area. Therefore, only the operation of setting the focus detection area in this embodiment will be described below. This embodiment will also be implemented using camera 100.

[0172] Figure 19 is a flowchart showing the focus detection area setting operation performed by the CPU 121 in this embodiment. In S3001, the CPU 121 obtains the image coordinates of the indicator (AF frame) of the unit focus detection area set by the user. As described in the first embodiment, the image coordinates of the unit focus detection area may be used instead of the indicator.

[0173] In S3002, the CPU 121 acquires information regarding the movement of the camera 100 from the output of the motion detection unit 142.

[0174] In S3003, the CPU 121 determines whether the camera 100 is being framed based on the movement of the camera 100 detected in S3002. If the CPU 121 determines that the camera 100 is being framed, it executes S3004; otherwise, it executes S3008. Since the determination of whether the movement of the camera 100 is a framing operation can be performed using a known method, a detailed explanation is omitted. The CPU 121 also determines the direction and amount of framing (for example, the amount of movement in the image coordinate system between frames) if it determines that a framing operation is taking place.

[0175] In S3008, the CPU 121 determines that since the camera 100 has not been framing, the photographer can easily align their intended subject with the currently set indicator. As a result, the CPU 121 sets the unit focus detection area corresponding to the currently set indicator as the final focus detection area and terminates the setting process. If there are multiple unit focus detection areas corresponding to the currently set indicator, S3007 is executed.

[0176] In S3004, CPU121 obtains the results of the subject detection and tracking process (subject detection information) performed in S302 in Figure 9 from RAM136. The subject detection information may include, for example, the position of the subject area (e.g., centroid coordinates), size, movement, and detection confidence level.

[0177] In S3005, the CPU 121 determines the reliability that the subject area from which information was acquired in S3004 is the subject intended by the photographer. This determination can be performed in the same way as the determination in S4002 in the first embodiment. If the CPU 121 determines that there is a high reliability that the subject intended by the photographer has been detected, it executes S3006; otherwise, it executes S3009.

[0178] In S3006, the CPU 121 detects the intended subject and sets the focus detection area during the framing operation. The CPU 121 expands the focus detection area so that it includes a unit focus detection area that overlaps with the subject area, at least in the framing direction. If the framing direction is horizontal, the CPU 121 can expand the focus detection area both to the right and to the left.

[0179] Furthermore, in S3006, the CPU 121 can correct the size of the focus detection area, which has been expanded in accordance with the movement of the camera 100. For example, in the direction in which a subject area that has been determined to be highly reliable as the subject intended by the photographer in S3005 exists, the CPU 121 corrects the focus detection area to further expand it or maintains the expansion. On the other hand, for other directions, the CPU 121 corrects the focus detection area to be relatively lower or does not expand at all.

[0180] As explained earlier, simply expanding the focus detection area increases the likelihood of background or unintended subjects being included in the focus detection area. Therefore, the focus detection area should not be expanded in directions where there is no subject, or the amount of expansion should be smaller than in directions where there is a subject.

[0181] In S3009, CPU121 expands the focus detection area according to the framing direction and amount of framing. When shooting while framing, misalignment between the subject and the indicator in the framing direction is likely to occur. For example, in horizontal framing, horizontal misalignment between the subject and the indicator is likely to occur. The magnitude of the misalignment is affected by the framing speed or acceleration.

[0182] Therefore, in S3009, where subject area information cannot be used, if a large misalignment between the indicator and the subject is expected based on the movement information of the camera 100, the focus detection area is expanded at least in the framing direction. This increases the probability that the subject is included in the focus detection area and suppresses the issue of focusing on the background when the indicator moves out of the subject. The magnification amount or magnification ratio can be determined such that it is larger when the framing amount is a second amount (>first amount) than when it is a first amount. The magnification amount or magnification ratio can be determined by storing a function or table in ROM 135 that associates the framing amount with the magnification amount or magnification ratio in advance and using it according to the framing amount. Note that the focus detection area may not be expanded when the framing amount is below a threshold.

[0183] S3009 is executed when no subject area with a detection confidence level above the threshold is detected, or when it is not located near the indicator. In such cases, the focus detection area is expanded at least in the framing direction to increase the likelihood that the subject is included in the focus detection area.

[0184] In S3007, CPU121 selects an appropriate unit focus detection area from the focus detection areas set in either S3006, S3008, or S3009. For example, CPU121 selects the unit focus detection area corresponding to the closest distance in the defocus map corresponding to the focus detection area. Alternatively, CPU121 selects the unit focus detection area closest to the center of the shooting range within the focus detection area. This is because subjects at close range or subjects close to the center of the shooting range within the focus detection area are more likely to be the subject intended by the photographer. Note that the unit focus detection area may be selected by other methods.

[0185] In this way, the size of the focus detection area is controlled depending on whether there is a high probability that the area of ​​the subject intended by the photographer has been detected, and whether or not framing is in progress. Therefore, it is possible to set a focus detection area of ​​an appropriate size according to the subject's situation and the photographer's framing skills, and even if the indicator moves away from the intended subject or the intended subject cannot be detected, the likelihood of proper focus control is increased.

[0186] Using Figures 20 to 27, a specific example of setting the focus detection area in this embodiment will be explained. Figure 20 schematically illustrates a scene in which a person swimming the butterfly stroke moves horizontally from left to right, with the user setting a single unit focus detection area where indicator I is displayed, and framing the camera 100 so that the subject is positioned within indicator I.

[0187] Figure 20 shows an example of framing by a highly skilled photographer. In Figures 20(a) and (b), the detected subject area M and the indicator I are in close agreement. Even in Figure 20(c), where the face is underwater and the subject area M is not detected, the indicator I is located on the subject's head. When framing is done as in Figure 20, there is a high probability that proper focus control will be achieved without enlarging the focus detection area.

[0188] On the other hand, Figure 21 shows an example of framing in the same scene as Figure 20, where the indicator I and the subject area M are misaligned. This is likely to occur when the photographer's framing skills are not high or when the subject's movement speed is not constant.

[0189] Figure 22 schematically illustrates a scene in which a subject performing a diving competition moves vertically from top to bottom, with the user setting a single unit focus detection area where indicator I is displayed, and framing the camera 100 so that the subject is positioned within indicator I.

[0190] Figure 22(a) shows an example of framing by a highly skilled photographer. Figure 22(b) shows an example of framing in the same scene as Figure 22(a), but with a misalignment between indicator I and the subject area M. Note that in Figure 22, the subject area is not detected because part of the face is obscured by the arm.

[0191] The discrepancy between indicator I and the subject area M (or intended area) is affected by the direction and speed of framing, as well as the subject's movement speed and acceleration. For example, horizontal framing results in a larger horizontal discrepancy. Similarly, fast subject movement and fast framing also increase the discrepancy. High subject acceleration makes framing more difficult, and therefore also increases the discrepancy.

[0192] Figures 23 and 24 show examples of setting the focus detection area in S3009 in a scene similar to those in Figures 20 and 21. Since the area of ​​the intended subject has not been detected, the focus detection area is set based on the movement (framing) information of the camera 100.

[0193] Here, horizontal framing is detected, and a focus detection region 1001 is set that is expanded horizontally (both left and right) around index I. Figure 23 shows a larger amount of framing than Figure 24, resulting in a larger magnification or magnification ratio than Figure 24. Specifically, in Figure 23, two unit focus detection regions A are added to the left and right, and a focus detection region 1001 containing five unit focus detection regions A is set. In Figure 24, one unit focus detection region A is added to the left and right, and a focus detection region 1001 containing three unit focus detection regions A is set.

[0194] Figures 25 and 26 show examples of focus detection area settings in S3006 in scenes similar to those in Figures 20 and 21. Since the area of ​​the intended subject has been detected, the focus detection area is expanded based on the movement (framing) information of the camera 100, and then the size of the focus detection area is corrected according to the positional relationship between indicator I and the subject area M.

[0195] First, because horizontal framing is detected, a focus detection area 1001 is provisionally set, which is enlarged by the same amount horizontally (both left and right) around index I. The amount or ratio of enlargement is set so that it includes one or more unit focus detection areas A that overlap with the subject area M. In the example in Figure 25, the distance between index I and the subject area M is greater than in the example in Figure 26, so the amount of horizontal enlargement of the focus detection area is also greater than in the example in Figure 26.

[0196] Subsequently, the image is corrected to reduce the magnification ratio or amount in the direction where the subject area M, which is detected as the area of ​​the subject intended by the photographer, does not exist, out of the two magnification directions directly opposite the index I. Figures 25 and 26 show an example in which the image is corrected so that it does not magnify in the framing direction (horizontal direction) to the right of index I where the subject area M does not exist.

[0197] Figure 27 shows an example of setting the focus detection area 1001 when the state of the subject changes from the state shown in Figure 25 or Figure 26, and the subject area M is no longer detected. Figure 27(a) shows the case when the subject's head (face) is hidden underwater, and Figure 27(b) shows the case when the subject's head (face) is hidden by a hand or splash of water.

[0198] In this case, the CPU 121 switches to the method of expanding the focus detection area based on the movement (framing) information of the camera 100, as explained with respect to Figure 23, and sets the focus detection area 1001.

[0199] Figures 20 to 27 illustrate examples of setting the focus detection area when the framing direction is horizontal, but the focus detection area can be set similarly when the framing direction is vertical or diagonal.

[0200] In this embodiment, if the area of ​​the subject that the photographer intends to capture is not detected, the focus detection area is expanded according to the framing direction and amount. Therefore, if the photographer unconsciously increases the framing amount because they cannot accurately predict the movement of the subject, the focus detection area will be expanded even further. This increases the likelihood that the subject will be included in the focus detection area and suppresses large changes in the subject's focus state.

[0201] Furthermore, if an area of ​​the subject that the photographer is presumably intending is detected, the focus detection area is enlarged to include the subject area, and the size of the focus detection area is corrected to prevent unnecessary enlargement. This makes it possible to increase the likelihood of focusing on the subject intended by the photographer while suppressing the possibility of focusing on the background or other subjects.

[0202] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0203] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0204] 100…Digital camera, 107…Image sensor, 121…CPU, 140…Subject detection unit 141...Dictionary data storage unit, 142...Motion detection unit

Claims

1. a detection means for detecting a subject area from a captured image; an acquisition means for acquiring the distance between a set focus detection area and a detected subject area; an enlarging means for enlarging the focus detection area when the subject area is not detected or when it is determined that the reliability that the detected subject area corresponds to the intended subject is low; The image pickup apparatus is characterized in that the enlarging means enlarges the focus detection area based on the distance previously acquired by the acquiring means.

2. 2. The image pickup apparatus according to claim 1, wherein the enlarging means enlarges the focus detection area based on the variation in the distance.

3. 3. The image pickup apparatus according to claim 2, wherein the focus detection area is enlarged based on a magnitude obtained by multiplying the magnitude of the variation in distance by a predetermined factor.

4. 2. The image pickup apparatus according to claim 1, wherein the enlarging means enlarges the focus detection area based on the representative value of the distance.

5. 5. The imaging device according to claim 4, wherein the enlarging means enlarges the focus detection area to a greater extent when the representative value of the distance is a second value greater than the first value.

6. 5. The image pickup apparatus according to claim 4, wherein the enlarging means enlarges the focus detection area based on the representative value of the distance and a framing amount of the image pickup apparatus.

7. 2. The image pickup apparatus according to claim 1, wherein said enlarging means enlarges said focus detection area significantly forward in the direction of movement of the subject.

8. 8. The imaging device according to claim 1, wherein the imaging device is capable of setting a plurality of unit focus detection areas, and the enlargement means enlarges the focus detection area so as to include the plurality of unit focus detection areas.

9. An imaging device, a motion detection means for detecting a motion of the imaging device; a determination means for determining whether the imaging device is undergoing a framing operation based on a movement of the imaging device; an enlarging unit that enlarges a focus detection area based on a movement of the imaging device when the determining unit determines that the imaging device is undergoing a framing operation; An imaging device comprising:

10. The camera further includes a detection means for detecting a subject area from the captured image, The imaging device according to claim 9, characterized in that, when the subject area is not detected or when it is determined that the reliability that the detected subject area corresponds to the intended subject is low, the enlargement means enlarges the focus detection area based on the framing direction and framing amount of the imaging device.

11. 11. The image pickup apparatus according to claim 10, wherein the enlarging unit enlarges the focus detection area in the framing direction of the image pickup apparatus by an enlargement amount or an enlargement ratio based on the framing amount.

12. The camera further includes a detection means for detecting a subject area from the captured image, If it is determined that the detected subject area corresponds to the intended subject with high confidence, the enlarging means enlarges the focus detection area in a framing direction of the imaging device, the amount of expansion of the focus detection area in the direction in which the subject area exists is smaller than the amount of expansion of the focus detection area in the direction in which the subject area does not exist; 10. The imaging device according to claim 9.

13. 13. The image pickup apparatus according to claim 12, wherein the enlarging unit enlarges the focus detection area in the framing direction so as to include the focus detection area.

14. An imaging device according to any one of claims 9 to 13, characterized in that the imaging device is capable of setting a plurality of unit focus detection areas, and the enlargement means enlarges the focus detection area to include one or more unit focus detection areas in addition to the currently set unit focus detection area.

15. A control method executed by an imaging device, comprising: Detecting a subject area from the captured image; Obtaining the distance between the set focus detection area and the detected subject area; and expanding the focus detection area when the subject area is not detected or when it is determined that the reliability that the detected subject area corresponds to the intended subject is low; The method for controlling an image pickup apparatus, wherein the enlarging includes enlarging the focus detection area based on the distance previously acquired.

16. A control method executed by an imaging device, comprising: detecting motion of the imaging device; determining whether the imaging device is performing a framing operation based on a movement of the imaging device; When it is determined that the imaging device is performing a framing operation, expanding a focus detection area based on the movement of the imaging device; 10. A method for controlling an imaging device, comprising:

17. A program for causing a computer included in an imaging apparatus to function as each of the means included in the imaging apparatus according to any one of claims 1 to 14.

Images