Imaging device, its control method, and program

Abstract

This method allows the photographer to focus on a specific part of a subject that is positioned at different depths, as desired. [Solution] The imaging device 100 includes a subject detection unit 140 that detects a predetermined subject from an image obtained by an imaging means and hierarchically detects characteristic regions of each subject, a system control unit 121 that selects two subjects from the predetermined subjects that are at different distances from the imaging device 100 in the depth direction, and a focus region determination unit 142 that determines a focus region from the regions hierarchically detected for each of the two subjects. The system control unit 121 sets an aperture value and a focal position that place the parts corresponding to the focus regions of the two selected subjects within the depth of field.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method thereof, and a program.

Background Art

[0002] Generally, when photographing a plurality of subjects having different positions in the depth direction or a subject long in the depth direction, the aperture of the lens device is often stopped down (the aperture value (F value) is increased) in order to widen the depth of field. In this method, while the depth of field is widened, if the aperture is stopped down too much, the blurring effect will be impaired. Also, when it is desired to clearly photograph a moving subject, it is necessary to increase the shutter speed (shorten the shutter speed), so there are cases where the aperture cannot be stopped down to a desired value in order to obtain the necessary exposure amount.

[0003] Therefore, Patent Document 1 proposes an imaging device equipped with a scene mode suitable for an imaging situation in which a plurality of moving subjects are imaging targets. Specifically, in the technique described in Patent Document 1, first, characteristic portions of a plurality of subjects detected from an image obtained by an imaging optical system are detected, and the position change in the depth direction thereof is compared between the latest image and the image immediately before it, thereby predicting the positions of the plurality of characteristic portions after a predetermined time has elapsed. Then, according to the predicted position information of the plurality of characteristic portions and the focal length of the imaging optical system, the depth of field that can bring the plurality of characteristic portions into a focused state within the image is set.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technology described in Patent Document 1, depth of field is set for subjects with different positions in the depth direction. However, in Patent Document 1, since multiple parts of the subject are not recognized hierarchically, it may not be possible to obtain a photograph that is reliably in focus on the part desired by the photographer.

[0006] This invention has been made in view of these circumstances, and aims to provide an imaging device that can obtain an image in focus on a part desired by the photographer when photographing multiple subjects that are positioned at different locations in the depth direction. [Means for solving the problem]

[0007] The imaging apparatus according to the present invention is characterized by comprising: subject detection means for detecting a predetermined subject from an image obtained by an imaging means including an imaging optical system; region detection means for hierarchically detecting characteristic regions of each of the predetermined subjects; selection means for selecting two subjects from among the predetermined subjects that are at different distances from the imaging means in the depth direction; determination means for determining a focus region from the regions hierarchically detected in each of the two subjects; and control means for setting an aperture value and focal position such that the portion of the two subjects corresponding to the focus region selected by the determination means is within the depth of field. [Effects of the Invention]

[0008] According to the present invention, when photographing multiple subjects at different positions in the depth direction, it is possible to obtain a photographic image that is in focus on the part desired by the photographer. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the schematic configuration of the imaging device according to this embodiment. [Figure 2] These are a plan view and a cross-sectional view illustrating the configuration of the image sensor in the imaging device. [Figure 3] This diagram illustrates the relationship between the image sensor and the exit pupil. [Figure 4]This is a flowchart of the imaging operation according to the first embodiment using the imaging device. [Figure 5] This is a flowchart of the subject determination process for S102. [Figure 6] This is a schematic diagram illustrating the hierarchical detection of feature regions in S201. [Figure 7] This is a flowchart of the aperture and focus position adjustment process for the S105. [Figure 8] This is a diagram illustrating an example of a filming scene. [Figure 9] This figure shows the relationship between the f-number, focal position, and depth of field in the shooting scene shown in Figure 8. [Figure 10] This is a flowchart of the imaging operation according to the second embodiment using the imaging device. [Figure 11] This figure shows another example of a filming scene. [Figure 12] This diagram illustrates the motion prediction processing in S605. [Figure 13] This is a flowchart of the motion prediction processing for S605. [Figure 14] This is a flowchart of the aperture and focus position adjustment process for the S607. [Figure 15] This diagram illustrates the relationship between the f-number and depth of field in the shooting scene shown in Figure 11. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Figure 1 is a block diagram showing a schematic configuration of the imaging device 100 according to an embodiment.

[0011] The imaging device 100 is a so-called digital still camera and has the function of capturing both still images and videos. The imaging device 100 has an imaging optical system consisting of a first lens group 101, an aperture 102, a second lens group 103, a third lens group 104, and an optical low-pass filter 106, as well as a zoom actuator 111, an aperture actuator 112, and a focus actuator 114.

[0012] The first lens group 101 is arranged to be movable along the imaging optical axis at the position closest to the subject (the front side of the imaging device 100) in the imaging optical system. The aperture 102 adjusts the amount of light passing through the imaging optical system by adjusting the aperture diameter. The second lens group 103 is integrated with the aperture 102 and is arranged to be movable in the direction of the imaging optical axis by an actuator (not shown), and cooperates with the first lens group 101 to realize a zoom function. The third lens group 104 is a so-called focus lens, and performs focus adjustment by moving in the direction of the imaging optical axis. The optical low-pass filter 106 is an optical element for reducing false colors and moiré in the captured image. The zoom actuator 111 is a driving means for moving the first lens group 101 in the direction of the imaging optical axis. The aperture actuator 112 drives the aperture 102 to adjust the aperture value (F value). The focus actuator 114 is a driving means for moving the third lens group 104 in the direction of the imaging optical axis.

[0013] The imaging device 100 includes an imaging element 107, a shutter 108, a system control unit 121, a strobe control unit 122, an auxiliary light driving unit 123, an imaging element driving unit 124, an image processing unit 125, a focus driving unit 126, an aperture driving unit 128, and a zoom driving unit 129. The imaging device 100 also includes a strobe device 115, an AF auxiliary light emitting unit 116, a display 131, an operation unit 132, a storage medium 133, a subject detection unit 140, a dictionary data storage unit 141, and a focus area determination unit 142.

[0014] The imaging element 107 is composed of a two-dimensional CMOS sensor and its peripheral circuit. The optical image formed on the imaging surface of the imaging element 107 after passing through the imaging optical system is converted into an analog electric signal by the imaging element 107, and the generated analog electric signal is output to the system control unit 121 via the imaging element driving unit 124. The system control unit 121 converts the analog electric signal into a digital image signal and supplies it to the required functional units. The shutter 108 controls the exposure time of the incident light with respect to the imaging element 107.

[0015] The system control unit 121 includes a CPU, ROM, RAM, A / D converter, D / A converter, communication interface circuit, etc., and comprehensively controls each part of the imaging device 100 by expanding a predetermined program stored in the ROM to the RAM. Note that some functions of the system control unit 121 may be implemented as a hardware circuit, and a reconfigurable circuit such as an FPGA may be used for some circuits. Also, in this embodiment, the system control unit 121 executes operations for focus detection described later, but a configuration may be adopted in which a dedicated hardware circuit executes a part of the operations in order to shorten the operation time.

[0016] The imaging element driving unit 124 controls the operation of the imaging element 107 according to commands from the system control unit 121. Note that an A / D conversion function for converting an analog electrical signal output from the imaging element 107 into a digital image signal may be provided in the imaging element driving unit 124 instead of the system control unit 121. The image processing unit 125 acquires a digital image signal from the system control unit 121, performs various image processes such as gamma conversion, color interpolation, compression / expansion, etc., and generates image data.

[0017] The display 131 has a liquid crystal display, an organic EL display, etc., and displays various images and information such as information regarding the imaging mode, live view video, confirmation image after shooting, focus state at the time of focus detection, etc. The operation unit 132 includes a power button, a release button, a zoom operation button, a mode selection button, a menu button, etc. Note that when the release button is half-pressed, a switch SW1 (not shown) is turned on, and shooting preparation processes such as AE (auto exposure) processing and AF (auto focus) are executed. Then, when the release button is fully pressed, a switch SW2 (not shown) is turned on, and a series of shooting operations from the exposure of the imaging element 107 to the storage of the image data of the captured image are executed.

[0018] The storage medium 133 is, for example, a memory card such as a flash memory that can be attached to the imaging device 100, and stores and saves image data of the captured image. The focus drive unit 126 drives the focus actuator 114 in response to a focus drive command from the system control unit 121, moving the third lens group 104 in the direction of the imaging optical axis. The aperture drive unit 128 drives the aperture actuator 112 in response to an aperture drive command from the system control unit 121, adjusting the aperture value of the aperture 102. The zoom drive unit 129 drives the zoom actuator 111 in response to the zoom operation of the photographer (user), moving the first lens group 101 in the direction of the imaging optical axis.

[0019] The subject detection unit 140 performs subject detection processing and feature region detection processing based on subject detection dictionary data set by the photographer of the imaging device 100 to detect a specific subject. The subject detection dictionary data defines the characteristics of each subject type (e.g., "person", "car", "animal", etc.) and is stored in the dictionary data storage unit 141. The subject detection processing detects the type of subject.

[0020] The operator of the imaging device 100 can set multiple subject detection dictionary data sets, rather than being limited to one, so that multiple types of subjects can be detected. When multiple subject detection dictionary data sets are set, the operator of the imaging device 100 can set from the operation unit 132 which order in which the set subject detection dictionary data sets will be used to detect subjects.

[0021] The dictionary data for subject detection further defines a higher level corresponding to the entire area of ​​the subject and a lower level corresponding to a part of that area, and there is not limited to just one lower level. For example, in the dictionary data for subject detection, "person (human)" has "whole body" as the higher level and multiple parts such as "upper body," "face (or head)," and "eyes" as the lower levels. Similarly, in the dictionary data for subject detection, "car" has "car body" as the higher level and "front grille," "headlights," "tires," etc. as the lower levels.

[0022] Feature region detection is a process that hierarchically detects characteristic parts of a detected subject in order to use them as the focus region. Specifically, each part of the higher and lower levels defined in the subject detection dictionary data is detected as a feature region. The feature region is detected as a rectangular area that encompasses the characteristic part of the subject, and preferably as the rectangular area with the smallest area.

[0023] Furthermore, the detection of the subject type and feature region can be done by first detecting the higher-level parts and then detecting the lower-level parts from the detected higher-level region, or vice versa. For example, when detecting a person (if a dictionary data for subject detection of "person" is set), detecting the lower-level "face" is easy. Therefore, after detecting the "face," the entire person can be detected (estimated) from the position of the "face," and then "eyes," "mouth," etc., can be detected from within the "face" region.

[0024] The focus region determination unit 142 determines one focus region for each subject from among the feature regions detected by the feature region detection process performed for each subject detected in the subject detection process.

[0025] Next, the pixel configuration of the image sensor 107 will be described. Figure 2(a) is a plan view showing the pixel arrangement of the image sensor 107. The z-axis shown in Figure 2(a) is parallel to the imaging optical axis (see Figure 1). The image sensor 107 has a roughly rectangular imaging surface in which a large number of 2x2 pixel groups 200 are arranged orthogonally to each other and in the x and y directions, respectively, and each pixel outputs an imaging signal. Note that only the 4x4 range is shown in Figure 2(a). Figure 2(b) is a plan view of the pixel 200G shown in Figure 2(a). Figure 2(c) is a cross-sectional view taken along arrow AA shown in Figure 2(b).

[0026] As shown in Figure 2(a), the 2x2 pixel group 200 includes one pixel 200R with R (red) spectral sensitivity, two pixels 200G with G (green) spectral sensitivity, and one pixel 200B with B (blue) spectral sensitivity, all arranged in a Bayer array. Pixels 200R, 200G, and 200B each have two focus detection pixels 201 and 202 arranged in a 2x1 array.

[0027] In this embodiment, a pixel consisting of two focus detection pixels arranged in the x-direction is used, but the embodiment is not limited to this. The two focus detection pixels may be arranged in the y-direction, and the number of focus detection pixels in a single pixel is not limited to two. The image sensor 107 may have a configuration in which a pixel having multiple focus detection pixels and an image generation pixel having one pixel are combined.

[0028] Since pixels 200G, 200B, and 200R have the same configuration except for their spectral characteristics, the following explanation will use pixel 200G as an example to describe its configuration in detail, and the explanations for pixels 200B and 200R will be omitted.

[0029] A microlens 305 for focusing incident light is provided on the light-receiving surface side of a semiconductor substrate, such as a silicon substrate, on which a 200G photodiode (PD) is formed. One microlens 305 is placed for each pixel, at a predetermined distance from the light-receiving surface in the z direction. In addition, one 200G pixel has two PDs (i.e., photoelectric conversion units 301 and 302) formed, which are divided into two in the x direction. The two photoelectric conversion units 301 and 302 correspond to the focus detection pixels 201 and 202 shown in Figure 2(a), respectively. Note that if each pixel is divided into Nx divisions in the x direction and Ny divisions in the y direction, then Nx × Ny = NLF (number of divisions) of photoelectric conversion units are formed.

[0030] The photoelectric conversion units 301 and 302 are each formed as pn junction photodiodes (PDs) consisting of a p-type layer and an n-type layer. Alternatively, they may be configured as pin-structured PDs with an intrinsic layer between the p-type and n-type layers.

[0031] In pixel 200G, a color filter 306 is provided between the microlens 305 and the photoelectric conversion units 301 and 302. The spectral transmittance of the color filter 306 may be changed for each pixel or each photoelectric conversion unit as needed, or a configuration without a color filter is also possible.

[0032] Light incident on pixel 200G is focused by microlens 305, spectrally separated by color filter 306, and then incident on photoelectric conversion units 301 and 302. In photoelectric conversion units 301 and 302, electron-hole pairs are generated according to the amount of incident light (amount of light received), and after they are separated in the depletion layer, the electrons (negative charge) are accumulated in the n-type layer. On the other hand, the holes (positive charge) are discharged to the outside of the image sensor 107 through the p-type layer connected to a constant voltage source (not shown). The electrons accumulated in the n-type layers of photoelectric conversion units 301 and 302 are transferred to the capacitance unit (FD) via a transfer gate, where they are converted into a voltage signal.

[0033] Figure 3 shows a cross-sectional view of the AA section of pixel 200G shown in Figure 2(b), viewed from the +y side, and the pupil plane at a predetermined distance in the z direction from the imaging surface of the image sensor 107. Note that in Figure 3, the x and y axes of the cross-sectional view are inverted compared to Figure 2 in order to correspond with the coordinate axes of the exit pupil plane.

[0034] The imaging surface of the image sensor 107 is positioned at the imaging plane of the imaging optical system. In other words, incident light is imaged onto the imaging surface. The two pupil division regions that constitute the pupil region 500, which can receive light from the entire pixel 200G when the photoelectric conversion units 301 and 302 are combined, are designated as the first pupil division region 501 and the second pupil division region 502, respectively. The first pupil division region 501 and the light-receiving surface of the photoelectric conversion unit 301, whose centroid is eccentric in the -x direction, are in a roughly conjugate relationship via the microlens 305. Therefore, the first pupil division region 501 corresponds to the pupil region that can receive light from the focus detection pixel 201, and its centroid is eccentric on the pupil surface towards the +x side. Similarly, the second pupil division region 502 and the light-receiving surface of the photoelectric conversion unit 302, whose centroid is eccentric in the +x direction, are in a roughly conjugate relationship via the microlens 305. Therefore, the second pupil division region 502 corresponds to the pupil region that can receive light with the focus detection pixel 202, and its centroid is eccentric to the -x side on the pupil plane.

[0035] Therefore, by performing a correlation operation on the output signals of focus detection pixels 201 and 202 to obtain phase difference information, and by converting the obtained phase difference information into a defocus amount (amount of focus shift) using known techniques, it is possible to perform image plane phase-detection autofocus to detect the focal position.

[0036] Furthermore, the imaging signal output from the image sensor 107 is used not only as a focus detection signal for performing on-sensor phase-detection autofocus, but also for generating the captured image and the photographed image. Although Figures 2 and 3 show a configuration in which one pixel is divided into two pupils in the x-direction, the number of pupil divisions and the direction of pupil division are not limited to these. For example, the pupil division direction may be only in the y-direction, or it may be divided in both the x and y directions (a total of four pupil divisions in 2x2).

[0037] <First Embodiment> Figure 4 is a flowchart of the imaging operation according to the first embodiment using the imaging device 100. Each process (step) indicated by the number S in this flowchart is realized by the system control unit 121, where the CPU loads a predetermined program stored in ROM into RAM and comprehensively controls the operation of each part of the imaging device 100. When the photographer turns on the power switch of the imaging device 100, the imaging operation by the image sensor 107 starts, a live view image is displayed on the display unit 131, and the process of S101 starts.

[0038] In S101, the system control unit 121 determines whether SW1 is turned on or not (whether the release button is half-pressed or not). If the system control unit 121 determines that SW1 is not turned on (NO in S101), it continues to execute the process in S101. If it determines that SW1 is turned on (YES in S101), it executes the process in S102.

[0039] In S102, the system control unit 121 detects a subject from the live view image (frame image) using the subject detection unit 140 and performs a process to determine the subject to focus on. The process in S102 will now be explained in detail with reference to Figure 5.

[0040] Figure 5 is a flowchart of the subject determination process in S102. In S201, the system control unit 121 performs a subject detection process to detect subjects from the frame image using the subject detection unit 140. The subject detection unit 140 uses the set subject detection dictionary data to detect subjects from the frame image that match the features defined in the subject detection dictionary data.

[0041] The subject detection unit 140 further performs the feature region detection process described above. Figure 6 is a schematic diagram illustrating the hierarchical detection of feature regions in S201. Figure 6(a) shows the case where the subject is a person, and Figure 6(b) shows the case where the subject is a car. Figure 6(a) schematically shows that a person 1001 is detected as the subject, and feature region 1002 corresponding to the "whole body" detected at the higher level, and feature regions 1003 and 1004 corresponding to the "face" and "eyes" detected at the lower level are detected. Figure 6(b) schematically shows that a car 1005 is detected as the subject, and feature region 1006 corresponding to the "car body" detected at the higher level, and feature region 1007 corresponding to the "front grille (front of the car body)" detected at the lower level are detected. Note that if a person is captured from behind, the "head," etc. may be detected at the lower level, and if a car is captured from the side, the "side window" or "tire," etc. may be detected at the lower level.

[0042] This invention relates to setting shooting parameters when multiple subjects are detected in the depth direction (depth of field direction) from the imaging device 100 toward the subject. Therefore, the following explanation will assume that multiple subjects are detected in the depth direction. If no multiple subjects are detected in the depth direction and only one subject is detected, the photographer can simply set the desired aperture value and take the picture.

[0043] In S202, the system control unit 121 selects one of the following subjects from among the multiple subjects detected in S201 as the first subject: the closest subject (the subject closest to the imaging device 100), the subject in the center of the image, or the subject with the largest area in the image. For example, if multiple people are detected, which person will be selected as the first subject may be automatically determined based on a pre-set, or it may be determined by the photographer's selection at the time of subject detection. For the sake of explanation, here we will assume that a close-up subject is selected as the first subject. The function of selecting the first subject and the second subject, which will be described later, may be provided to the subject detection unit 140.

[0044] In S203, the system control unit 121, using the focus area determination unit 142, determines one area to focus on (focus area) from among the feature areas detected hierarchically for the first subject. In Figure 6(a), one of feature areas 1002, 1003, or 1004 is determined as the focus area, while in Figure 6(b), one of feature areas 1006 or 1007 is determined as the focus area. The determined focus area is displayed overlaid on the live view image shown on the display unit 131.

[0045] Furthermore, if multiple feature regions are detected, any of them may be selected as the focus region. However, rules may be predetermined, such as prioritizing lower-level feature regions regardless of the type of subject.

[0046] In S204, the system control unit 121 selects a subject that is further away than the first subject (a distant subject) from among the multiple subjects detected in S201 as the second subject. Consequently, the positional relationship between the first subject and the second subject is that of a near-view subject and a distant subject. Even if the central subject or the subject with the largest area on the image plane was selected as the first subject in S202, the second subject must be a subject that is further away in the depth direction than the first subject. In other words, the first subject is selected on the premise that the second subject is located further away.

[0047] In S205, the system control unit 121 determines the focus region from the feature regions detected hierarchically for the second subject using the focus region determination unit 142. Note that the processing in S205 is performed in the same way as in S203, so the explanation is omitted here.

[0048] In S206, the system control unit 121 calculates the depth difference (distance in the depth direction) between the first subject and the second subject, and terminates this process. The calculation method can be any known technique. This allows the process to proceed from S102 to S103 in the flowchart of Figure 4.

[0049] Returning to the explanation of the flowchart in Figure 4, in S103, the system control unit 121 controls the focus drive unit 126 and the aperture drive unit 128 to open the aperture 102 to its maximum (minimum F value) and executes the AF operation for the first subject using the focus area determined in S203.

[0050] In S104, the system control unit 121 determines whether or not a focus state has been achieved for the first subject by the AF operation in S103. Specifically, the system control unit 121 determines the amount of defocus from the output signals of the focus detection pixels 201 and 202 (photoelectric conversion units 301 and 302), and determines that a focus state is achieved if the determined amount of defocus is within a preset range. If the system control unit 121 determines that a focus state has not been achieved (NO in S104), it executes the process in S102, and if it determines that a focus state has been achieved (YES in S104), it executes the process in S105.

[0051] In S105, the system control unit 121 sets an appropriate depth of field by adjusting the aperture and focal position using the previously determined defocus amount. The process in S105 will now be explained with reference to Figure 7. Figure 7 is a flowchart of the aperture and focal position adjustment process in S105.

[0052] In S103, the aperture 102 is open (the F-number is set to the minimum value), so in S301, the system control unit 121 closes the aperture 102 to a predetermined F-number (increases the F-number). The "predetermined aperture value" is set according to the configuration of the imaging optical system, and can be, for example, the upper limit of the F-number range desired by the photographer, and can be set by the photographer from the operation unit 132.

[0053] In S302, the system control unit 121 determines whether the first subject and the second subject are within the depth of field. If the system control unit 121 determines that the first subject and the second subject are within the depth of field (YES in S302), it executes the process in S303.

[0054] In S303, the system control unit 121 opens the aperture 102 by a certain amount (decreases the F value by a certain amount). In S304, the system control unit 121 moves the focal position toward the distant side (towards the second subject). In S305, the system control unit 121 determines whether the first subject and the second subject are within the depth of field. In other words, in S303 to S305, the system control unit 121 determines whether the state in which the first subject and the second subject are within the depth of field is maintained while gradually driving the aperture 102 toward the open side and shifting the focal position toward the second subject. If the system control unit 121 determines that the first subject and the second subject are within the depth of field (YES in S305), it executes the process in S303; if it determines that they are not within the depth of field (NO in S305), it executes the process in S306.

[0055] In S306, the system control unit 121 sets the aperture value and focal position to the values ​​used in the 'YES' determination immediately before the 'NO' determination in S305 as the shooting conditions. If the aperture value and focal position become such that the first and second subjects are no longer within the depth of field, the system can determine the minimum F value that keeps the first and second subjects within the depth of field by returning to the previous aperture value and focal position.

[0056] In S307, the system control unit 121 checks whether the first subject and the second subject are in focus. For example, if there is no change in the relative positions of the subjects and the photographer, the first subject and the second subject should be within the depth of field in S306, so the determination in S307 is 'YES'. On the other hand, if the first subject or the second subject moves, there is no guarantee that the first subject or the second subject will be within the depth of field when the process in S306 is performed. If the system control unit 121 determines that the first subject and the second subject are in focus (YES in S307), it terminates the processing in this flowchart and executes the process in S106. On the other hand, if the system control unit 121 determines that the first subject or the second subject are not in focus (NO in S307), it terminates the processing in this flowchart and executes the process in S101.

[0057] If the system control unit 121 determines in S302 that the first subject and the second subject are not within the depth of field (NO in S302), it executes the process in S308.

[0058] In S308, the system control unit 121 returns the aperture 102 to its widest setting, which was the aperture value before processing in S301, refocuses on the first subject, and then executes the process in S106. If the determination in S302 is 'NO', and the specified aperture value set in S301 is not the maximum value, that is, if the aperture 102 can be stopped down further, the system may first change the F value to the maximum value and then execute the process in S302 again. This method can be used when the photographer does not impose any restrictions on the F value during shooting. As a result, if the determination in S302 again is 'NO', the process in S308 may be executed, and if it is 'YES', the process in S303 may be executed.

[0059] Furthermore, in the S105 process, it is possible to determine the minimum F-number within which both the first and second subjects are within the depth of field by gradually stopping down the aperture 102 from the wide-open state in S301 while shifting the focal position toward the second subject. However, it is easier to control to determine the minimum F-number within which both the first and second subjects are within the depth of field starting from a state where the aperture 102 is stopped down.

[0060] Returning to the explanation of the flowchart in Figure 4, in S106, the system control unit 121 sets the shutter speed Tv and ISO sensitivity that will result in proper exposure with the aperture value set in S105.

[0061] In S107, the system control unit 121 determines whether SW2 is turned on or not (whether the release button is fully pressed or not). If the system control unit 121 determines that SW2 is turned on (YES in S107), it executes the process in S108. If it determines that SW2 is not turned on (NO in S107), it executes the process in S109.

[0062] In S108, the system control unit 121 performs the shooting process (a series of processes from exposure to the image sensor 107 to saving the image data), and then terminates the process according to this flowchart.

[0063] In S109, the system control unit 121 determines whether SW1 is ON (whether the release button is half-pressed). If the system control unit 121 determines that SW1 is not ON (NO in S109), it terminates the process according to this flowchart. If it determines that SW1 is ON (YES in S109), it executes the process in S107.

[0064] Figure 8(a) shows an example of a shooting scene in the first embodiment, and Figure 8(b) shows the positional relationship between the imaging device 100 and the subject in the shooting scene of Figure 8(a). In the shooting scene of Figure 8(a), the subject detection unit 140 detects three people, and the system control unit 121 or the photographer selects the person 1101 closest to the camera as the first subject and the person 1102 furthest from the camera as the second subject. The focus area determination unit 142 determines the focus area from the feature areas detected hierarchically for the first subject person 1101 and the second subject person 1102. Here, since pupils are detected in people 1101 and 1102, the focus can be set to the lower-level "pupil" for the first subject and the second subject. In this case, AF in S103 is performed on the feature area 1105 of the first subject (person 1101). Then, in S302 and S305, it is determined whether or not the feature regions 1105 of person 1101 and 1106 of person 1102 are within the depth of field.

[0065] Figure 9 illustrates the relationship between the F-number, focal position, and depth of field in the shooting scene shown in Figure 8. The front depth of field Lf and the rear depth of field Lr are expressed by the following equations 1 and 2, respectively, using the distance L from the image sensor to the first subject, the focal length f of the imaging optical system, the acceptable circle of confusion δ, and the aperture value F. Since the range in focus is from 'L-Lf' to 'L+Lr', the depth of field range ΔL is expressed by the following equation 3.

[0066]

number

[0067] In Figure 9, the pupil of the first subject (focus area) is at position '0cm', and the pupil of the second subject is at position '70cm'. When the focus is set to the pupil of the first subject, with a focal length f=50mm of the imaging optical system, a distance L=3m from the imaging plane of the image sensor 107 to the pupil of the first subject, and an aperture value of F=8, the rear depth of field Lr is approximately 71cm. The allowable circle of confusion δ is a value predetermined by the size of the image sensor 107. Therefore, in this state, it is possible to achieve focus on the pupils of both the first and second subjects within the range of the rear depth of field Lr.

[0068] At an aperture of F=5.6, the rear depth of field Lr is approximately 53cm, making it impossible to achieve focus on the eyes of both the first and second subjects using only the rear depth of field Lr. Therefore, the front depth of field Lf = approximately 40cm is utilized. For example, by shifting the focal point 18cm towards the far side, it is possible to achieve focus on the eyes of both the first and second subjects even at an aperture of F=5.6.

[0069] As described above, in the shooting operation according to the first embodiment, when multiple subjects at different positions in the depth direction are detected, the multiple detected subjects are captured within the depth of field with the aperture opened as wide as possible. At this time, a hierarchical feature region is detected for each of the multiple subjects, and the focus region is determined from among them. As a result, it is possible to obtain a high-quality image that is in focus on the part desired by the photographer and has an appropriate amount of bokeh.

[0070] <Second Embodiment> While the first embodiment focused on a subject with little movement, the second embodiment describes a case where the subject is moving.

[0071] Figure 10 is a flowchart of the shooting operation according to the second embodiment of the imaging device 100. Each process (step) indicated by the number S in this flowchart is realized by the system control unit 121, where the CPU loads a predetermined program stored in ROM into RAM and comprehensively controls the operation of each part of the imaging device 100. The photographer turns on the power switch of the imaging device 100, sets the still image shooting mode, and sets the AF mode to Servo AF (subject tracking AF). After these settings, the imaging operation by the image sensor 107 starts, the live view image is displayed on the display unit 131, and the process of S601 starts.

[0072] The processing in S601 to S603 is equivalent to the processing in S101 to S103 in the first embodiment, so its explanation is omitted. Figure 11(a) shows an example of a shooting scene in the second embodiment, and the positional relationship between the imaging device 100 and the subject in the shooting scene in Figure 11(a) is shown in Figure 11(b). In the shooting scene in Figure 11(a), in the subject determination processing in S602, the subject detection unit 140 detects two cars 1301 and 1302, and the system control unit 121 selects the close-up car 1301 as the first subject and the distant car 1302 as the second subject. In this case, the focus area determination unit 142 determines the feature area 1311 of the lower-level tip of the close-up car 1301 as the focus area. On the other hand, the focus area determination unit 142 determines the entire feature area 1312 as the focus area because, regarding the distant car 1302, its tip is obscured by the car 1301 and cannot be constantly detected.

[0073] At S604, the system control unit 121 starts subject tracking using servo AF. Subject tracking here is performed using the focus area set in S602 as a template, and a known template matching process, specifically, a process of finding image areas similar to the template from successively obtained frame images.

[0074] In S605, the system control unit 121 performs a motion prediction calculation to predict the position of the subject in the depth direction after a predetermined time has elapsed, based on the subject image plane position obtained in the previous S603 and the amount of defocus obtained from the most recent focus detection, and performs focus control based on the calculation result. The process in S605 will now be explained in detail.

[0075] Figure 12 is a diagram illustrating the motion prediction process in S605. Figure 13 is a flowchart of the motion prediction process in S605. In S701, the system control unit 121 refers to the historical data of the image plane position of the first subject and the second subject. Here, the image plane position is the focal position when the imaging optical system (lens group) is in focus with respect to the subject. Figure 12(a) shows the historical data of the image plane position of the subject. Note that in the explanation referring to Figure 12, the subject is not limited to the first subject or the second subject, but is a general explanation. In Figure 12(a), the historical data of the image plane position of the subject represents the change in the image plane position 1201 of the subject acquired at predetermined time intervals in the past.

[0076] In S702, the system control unit 121 references the current image plane positions of the first and second subjects. In S703, the system control unit 121 predicts the focal position of the first and second subjects after a predetermined time based on historical data. In S704, the system control unit 121 updates the prediction curves for the focal positions of the first and second subjects. Figure 12(a) shows the prediction curve 1202 for the focal position based on historical data of the image plane position 1201.

[0077] Prediction curve 1202 uses a pre-defined function. For example, a quadratic function f(t) shown in equation 4 below, where time t is the variable, can be used. In the S704 prediction curve update, the values ​​of the coefficients a, b, and c of the quadratic function f(t) are updated.

[0078]

number

[0079] In S705, the system control unit 121 determines whether the prediction accuracy is within a predetermined range. Prediction accuracy is a value that indicates the degree of agreement between the focal position and the prediction curve, and values ​​such as the coefficient of determination or mean square error (known techniques) can be used. Figure 12(b) is a diagram illustrating historical data when the prediction accuracy is high, and Figure 12(c) is a diagram illustrating historical data when the prediction accuracy is low.

[0080] If the system control unit 121 determines that the prediction accuracy is within a specified range (for example, the relationship shown in Figure 12(b)) (YES in S705), it executes the process in S706. In S706, the system control unit 121 updates the predicted focal positions of the first subject and the second subject, ending the process in this flowchart and proceeding to S606.

[0081] In the determination process of S705, if the system control unit 121 determines that the prediction accuracy is not within the range of the specified value (for example, the relationship shown in Figure 12(c)) (NO in S705), it executes the process of S707. In S707, the system control unit 121 focuses on the first subject and terminates the process of this flowchart, then proceeds to S608.

[0082] Let's return to the explanation of the flowchart in Figure 10. In S606, the system control unit 121 determines whether or not a focus state has been achieved for the first subject. If the system control unit 121 determines that a focus state has been achieved for the first subject (YES in S606), it executes the process in S607. If it determines that a focus state has not been achieved for the first subject (NO in S606), it executes the process in S602.

[0083] In S607, the system control unit 121 performs adjustment processing for the aperture 102 and the focal position. Figure 14 is a flowchart of the aperture and focal position adjustment processing in S607. The processing in S801 to S806 is the same as the processing in S301 to S306 of the flowchart in Figure 7 described in the first embodiment, so the explanation is omitted here.

[0084] In S807, the system control unit 121 determines whether the predicted positions of the first and second subjects are within the focus movement range of the imaging optical system (the range from the shortest shooting distance to infinity). This allows it to check whether or not it is impossible to focus on the first and second subjects. If the system control unit 121 determines that the positions are within the focus movement range (YES in S807), it executes the process in S808; if it determines that the positions are outside the focus movement range (NO in S807), it executes the process in S602.

[0085] In S808, the system control unit 121 performs a control to add the defocus amount obtained from motion prediction to the focal position set in S806 to determine the focal position to be used for shooting, thereby ending the processing of this flowchart and executing the process in S608. The "defocus amount obtained from motion prediction" is the difference between the focus position at time t and the focus position at time t+Δt, which is time Δt after time t. Referring to Figure 12(a), it is expressed as the difference between the focal position at the "prediction" time and the latest subject image plane position in the history data.

[0086] Returning to the explanation of the flowchart in Figure 10, the processes from S608 to S611 are the same as the processes from S106 to S109 in the first embodiment, so the explanation will be omitted.

[0087] Figure 15 illustrates the relationship between the F-number and depth of field in the shooting scene shown in Figure 11. Assuming that both cars 1301 and 1302 are approximately 2m wide and 4.5m long, in a situation where the two cars 1301 and 1302 are competing, the required depth of field is approximately three times the total length (approximately 13.5m). When the focus is set on the front of the close-up car 1301, the aperture value required to keep cars 1301 and 1302 within the rear depth of field is F36. The difference in depth of field between cars 1301 and 1302 can be obtained by adjusting the aperture and focal position in S607 (processing according to the flowchart in Figure 14). For example, the focal position can be moved 6m from the front of car 1301 towards the background. With an aperture value of F18, the depth of field in front and behind the subject becomes 6.3m and 8.1m respectively (totaling 14.4m), allowing cars 1301 and 1302 to be captured within the depth of field.

[0088] In the shooting operation according to the second embodiment described above, even if multiple moving objects are detected as subjects in the depth direction, the positions of the multiple moving objects are predicted and kept within the depth of field while shooting is performed with the aperture opened as wide as possible. At this time, hierarchical feature regions are detected for each of the multiple subjects and the focus region is determined from among them. This makes it possible to obtain a high-quality image that is in focus on the part desired by the photographer. In addition, by opening the aperture as wide as possible, the shutter speed can be increased, so an image can be obtained that has a moderate amount of bokeh and reduced blurring of the subject (the subject is captured clearly).

[0089] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Furthermore, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0090] For example, although the present invention was described as a digital still camera in the above embodiment, it is not limited to this, and the present invention can be applied to electronic devices equipped with an imaging function that acquires an optical image of a subject obtained by an imaging optical system as image data using an image sensor. Examples of such camera-equipped electronic devices include smartphones and other camera-equipped portable communication terminals, as well as tablet PCs.

[0091] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0092] This embodiment includes the following configurations, methods, and programs. (Configuration 1) An imaging device comprising: subject detection means for detecting a predetermined subject from an image obtained by an imaging means including an imaging optical system; region detection means for hierarchically detecting characteristic regions of each of the predetermined subjects; selection means for selecting two subjects from among the predetermined subjects that are at different distances from the imaging means in the depth direction; determination means for determining a focus region from the regions hierarchically detected in each of the two subjects; and control means for setting an aperture value and focal position so that the portion of the two subjects corresponding to the focus region selected by the determination means is within the depth of field. (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that the selection means selects the close-up subject from among the subjects detected by the subject detection means, the subject located in the center of the image, and the subject with the largest area in the image, to be the close-up subject of the two subjects. (Configuration 3) The imaging apparatus according to Configuration 2, wherein the imaging optical system has a lens and an aperture, and the control means sets the aperture to its widest value to focus on the close-up subject, and then, with the aperture stopped down to a predetermined aperture value, determines whether the two subjects are within the depth of field. (Configuration 4) The imaging device according to Configuration 3, characterized in that when the control means determines that the two subjects are not within the depth of field when the aperture is set to the predetermined aperture value, it sets the aperture to its widest value and focuses on the closer of the two subjects. (Configuration 5) The imaging device according to Configuration 3 or 4, characterized in that when the control means determines that the two subjects are within the depth of field when the aperture is set to the predetermined aperture value, it drives the aperture to the open side and shifts the focal position toward the distant side to determine whether or not the two subjects are within the depth of field. (Configuration 6) The imaging device according to Configuration 5, characterized in that when the control means determines that the two subjects do not fall within the depth of field range, it sets the aperture value and focal length to the values ​​used when the two subjects immediately preceding the determination that they fall within the depth of field range as the shooting conditions. (Configuration 7) The imaging device according to any one of Configurations 4 to 6, characterized in that the control means sets the shutter speed and ISO sensitivity that can obtain proper exposure during shooting using the set aperture value. (Configuration 8) An imaging apparatus according to any one of Configurations 1 to 7, characterized in that it includes a prediction means for predicting the positions of the two subjects in the depth direction after a predetermined time has elapsed, based on the subject image plane position obtained by past focus detection for the two subjects and the amount of defocus obtained by the most recent focus detection. (Configuration 9) The imaging apparatus according to Configuration 8, wherein the control means focuses on the closer of the two subjects when the accuracy of the predicted positions in the depth direction of the two subjects obtained by the prediction means is not within a predetermined range of values. (Configuration 10) An imaging device according to any one of Configurations 1 to 9, comprising storage means for storing dictionary data in which a specific subject type and a higher hierarchy corresponding to the whole of the specific subject and a lower hierarchy corresponding to a part of the subject are defined, wherein the subject detection means detects a subject of the type defined in the dictionary data, and the region detection means detects from the image a region including the whole of the specific subject defined as the higher hierarchy and a region including the part defined as the lower hierarchy as characteristic regions. (Configuration 11) An imaging device comprising: a subject detection means for detecting a predetermined subject from an image obtained by an imaging means including an imaging optical system; a region detection means for hierarchically detecting characteristic regions of each of the predetermined subjects; a selection means for selecting two subjects from among the predetermined subjects that are at different distances from the imaging means in the depth direction; a determination means for determining a focus region from the regions hierarchically detected for each of the two subjects; and a control means for shifting the focal position when focusing on the focus region of the near subject towards the far subject, and determining the minimum aperture value such that the two subjects are within the depth of field. (Method 1) A method for controlling an imaging device, comprising the steps of: detecting a predetermined subject from an image obtained by an imaging means; hierarchically detecting characteristic regions of each of the predetermined subjects; selecting two subjects from among the predetermined subjects that are at different distances from the imaging means in the depth direction; determining a focus region from the regions hierarchically detected for each of the two subjects; and setting an aperture value and a focal position that bring the portion corresponding to the determined focus region of each of the two subjects within the depth of field. (Method 2) A method for controlling an imaging device, comprising the steps of: detecting a predetermined subject from an image obtained by an imaging means; hierarchically detecting characteristic regions of each of the predetermined subjects; selecting two subjects from among the predetermined subjects that are at different distances from the imaging means in the depth direction; determining a focus region from among the regions hierarchically detected for each of the two subjects; and shifting the focal position when focusing on the focus region of the near subject towards the far subject to find the minimum aperture value such that the two subjects are within the depth of field. (Program 1) A program that causes a computer to execute each step of the control method for the imaging device described in Method 1 or 2. [Explanation of Symbols]

[0093] 100 Imaging device 102 aperture 107 Image sensor 121 System Control Unit 140 Subject detection unit 142 Focus area determination unit

Claims

1. A subject detection means for detecting a predetermined subject from an image obtained by an imaging means including an imaging optical system, Region detection means for hierarchically detecting characteristic regions of each of the predetermined subjects, A selection means for selecting two subjects from the predetermined subjects that are at different distances from the imaging means in the depth direction, A determination means for determining a focus region from among the regions detected hierarchically in each of the two subjects, An imaging device comprising: a control means for setting an aperture value and focal position such that the portion corresponding to the focus region selected by the determination means for the two subjects is placed within the depth of field range.

2. The imaging apparatus according to claim 1, characterized in that the selection means selects the close-up subject from among the subjects detected by the subject detection means, the subject located in the center of the image, and the subject with the largest area in the image, to be the close-up subject of the two subjects.

3. The aforementioned OTO optical system has a lens and an aperture, The imaging apparatus according to claim 2, characterized in that the control means sets the aperture to its widest value to focus on the close-up subject, and then, with the aperture stopped down to a predetermined aperture value, determines whether the two subjects are within the depth of field.

4. The imaging apparatus according to claim 3, characterized in that, when the control means determines that the two subjects are not within the depth of field when the aperture is set to the predetermined aperture value, it sets the aperture to its widest value and focuses on the closer of the two subjects.

5. The imaging apparatus according to claim 3 or 4, characterized in that when the control means determines that the two subjects are within the depth of field when the aperture is set to the predetermined aperture value, it drives the aperture to the open side and shifts the focal position toward the distant side to determine whether or not the two subjects are within the depth of field.

6. The imaging apparatus according to claim 5, characterized in that, when the control means determines that the two subjects are not within the depth of field, it sets the aperture value and focal length to the values ​​used when the two subjects were within the depth of field immediately prior to that determination as the shooting conditions.

7. The imaging device according to claim 6, characterized in that the control means sets the shutter speed and ISO sensitivity that can obtain proper exposure during shooting using the set aperture value.

8. The imaging apparatus according to claim 1 or 2, further comprising a prediction means for predicting the positions of the two subjects in the depth direction after a predetermined time has elapsed, based on the subject image plane position obtained by past focus detection for the two subjects and the amount of defocus obtained by the most recent focus detection.

9. The imaging apparatus according to claim 8, wherein the control means focuses on the closer of the two subjects when the accuracy of the predicted positions in the depth direction of the two subjects obtained by the prediction means is not within a predetermined range of values.

10. The system includes a storage means that stores dictionary data in which a specific subject type is defined, and a higher hierarchy corresponding to the whole of the specific subject and a lower hierarchy corresponding to a part of it are defined. The subject detection means detects subjects of the type defined in the dictionary data, The imaging apparatus according to claim 1 or 2, characterized in that the region detection means detects from the image a region including the entirety of the specific subject defined as the higher layer, and a region including the part defined as the lower layer, as characteristic regions.

11. A subject detection means for detecting a predetermined subject from an image obtained by an imaging means including an imaging optical system, Region detection means for hierarchically detecting characteristic regions of each of the predetermined subjects, A selection means for selecting two subjects from the predetermined subjects that are at different distances from the imaging means in the depth direction, A determination means for determining a focus region from among the regions detected hierarchically in each of the two subjects, An imaging device characterized by comprising: a control means for shifting the focal position when focusing on the focus area of ​​the near-view subject of the two subjects toward the far-view subject, and determining the minimum aperture value such that both subjects are within the depth of field.

12. A method for controlling an imaging device, A step of detecting a predetermined subject from an image obtained by an imaging means, The steps include: detecting characteristic regions of each of the predetermined subjects in a hierarchical manner; The steps include selecting two subjects from the predetermined subjects that are at different distances from the imaging means in the depth direction, The steps include determining the focus region from the regions detected hierarchically in each of the two subjects, A method for controlling an imaging device, characterized by comprising the steps of setting an aperture value and a focal position such that the portion corresponding to the determined focus area for each of the two subjects is placed within the depth of field.

13. A method for controlling an imaging device, A step of detecting a predetermined subject from an image obtained by an imaging means, The steps include: detecting characteristic regions of each of the predetermined subjects in a hierarchical manner; The steps include selecting two subjects from the predetermined subjects that are at different distances from the imaging means in the depth direction, The steps include determining the focus region from the regions detected hierarchically in each of the two subjects, A method for controlling an imaging device, characterized by comprising the step of shifting the focal position when focusing on the focus area of ​​the closer subject of the two subjects toward the farther subject, and determining the minimum aperture value such that both subjects are within the depth of field.

14. A program that causes a computer to execute each step of the control method for an imaging device according to claim 12 or claim 13.

Images