Imaging device, its control method, and program

Abstract

The present invention provides an imaging device capable of acquiring still images and videos with suppressed degradation in quality, even when the depth of field difference or the time-dependent change in the center of focus of multiple subjects is oscillating. [Solution] The imaging device comprises a calculation means for calculating depth difference information between multiple subjects, and a depth control means for controlling the aperture based on the depth difference information, wherein the depth control means switches the aperture control method depending on whether the time change of the depth difference information is in a vibrating state or not. In another embodiment, the imaging device comprises an acquisition means for acquiring position information in the depth direction of multiple subjects, and a focus control means for controlling the focus position based on the position information in the depth direction, wherein the focus control means switches the focus position control method depending on whether the time change of the center of focus position based on the position information in the depth direction is in a vibrating state or not.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method thereof, and a program.

Background Art

[0002] Generally, when photographing a plurality of subjects having different positions in the depth direction or a subject long in the depth direction, the aperture of the lens device is often narrowed (the aperture value (F value) is increased) in order to widen the depth of field. Further, a technique for controlling the aperture of the imaging optical system based on the position information in the depth direction so that a plurality of dynamic subjects are within a predetermined depth of field has been disclosed (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When photographing a plurality of moving subjects within the depth of field, each subject does not necessarily move in the same way in the focusing direction. In the technique described in Patent Document 1 above, the focus target distance is determined based on the information of the trailing prediction distance and the leading prediction distance so that a plurality of dynamic subjects are within a predetermined depth of field, and then the aperture value at which the leading and trailing ends are within the range is determined. Therefore, when the focus target distance or the aperture value changes vibrantly, the depth difference between the plurality of subjects or the temporal change of the focus position becomes vibratory, and there is a concern that the quality of the acquired still image or moving image may deteriorate due to focus tracking or luminance flickering.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide an imaging device capable of acquiring a still image or a moving image with suppressed quality degradation even when the depth difference between a plurality of subjects or the temporal change of the focus center position is vibratory. [Means for solving the problem]

[0006] To solve the above problems, according to one aspect of the present invention, the imaging device comprises a calculation means for calculating depth difference information between multiple subjects, and a depth control means for controlling the aperture based on the depth difference information, wherein the depth control means switches the aperture control method depending on whether the time change of the depth difference information is in a vibrating state or not. Furthermore, according to another aspect of the present invention, the imaging device comprises an acquisition means for acquiring position information in the depth direction of multiple subjects, and a focus control means for controlling the focus position based on the position information in the depth direction, wherein the focus control means switches the focus position control method depending on whether the time change of the center of focus position based on the position information in the depth direction is in a vibrating state or not. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an imaging device that can acquire still images and videos with suppressed degradation of quality, even when the difference in depth of field or the time change in the center position of focus of multiple subjects is oscillating. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of digital camera 100. [Figure 2] This is a block diagram showing the electrical configuration of the Digital Camera 100. [Figure 3] This is a conceptual diagram showing an example of a pixel arrangement in an image sensor. [Figure 4] These are schematic plan views and schematic cross-sectional views of a pixel. [Figure 5] This is a diagram showing the focus detection region. [Figure 6] This is a flowchart showing the focus detection process. [Figure 7] This figure shows the state in which the primary subject 701 and secondary subject 702 have been detected. [Figure 8]This figure shows the state in which the main subject 801, secondary subject 802, and secondary subject 803 are detected. [Figure 9] This is an illustrative diagram showing the time-dependent change in the amount of defocus for each subject. [Figure 10] This is an illustrative diagram showing the difference in depth of field between subjects and the change in focus position over time. [Figure 11] This is an illustrative diagram showing the time evolution of the depth of field difference and the acceleration of the focus position for each subject. [Figure 12] This is a flowchart showing the process of switching between depth control and focus control modes. [Figure 13] This figure shows an example of the camera's menu screen for selecting a shooting mode. [Modes for carrying out the invention]

[0009] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] In the following embodiments, the present invention will be described in relation to cases where it is implemented using an imaging device such as a digital camera. However, an imaging function is not essential to the present invention, and it can be implemented with any imaging device. Such imaging devices include mobile phones, smartphones, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented with other imaging devices.

[0011] <Overall Structure> Figure 1 shows an example of the configuration of a digital camera (hereinafter also simply referred to as "camera") 100 as an example of the imaging device of the present invention. Figure 2 is a block diagram showing the electrical configuration of the camera 100 in Figure 1.

[0012] As shown in FIG. 1, on the front side (subject side) of the camera body 101, a detachable and replaceable lens unit 120 is mounted. The lens unit 120 has a focus lens 121, an aperture 122, etc., and is electrically connected to the camera body 101 via a mount contact portion 123. Thereby, it is possible to adjust the amount of light taken into the camera body 101 and the focus position. Note that the focus lens 121 can also be manually adjusted by the user.

[0013] The image pickup device 104 is composed of a CMOS sensor or the like and includes an infrared cut filter, a low-pass filter, etc. The image pickup device 104 photoelectrically converts the subject image formed by passing through the imaging optical system of the lens unit 120 during shooting, and transmits a signal for generating a photographed image to the arithmetic unit 102. The arithmetic unit 102 generates a photographed image from the received signal, stores it in the image storage unit 107, and displays it on the display unit 105 such as an LCD. The shutter 103 shields the image pickup device 104 during non-shooting, and opens during shooting to expose the image pickup device 104.

[0014] Next, the configuration related to control will be described using FIG. 2. The arithmetic unit 102 includes a multi-core CPU, a RAM, and a ROM capable of parallel processing a plurality of tasks, as well as a dedicated circuit for executing specific arithmetic processing at high speed. By these hardware components, the arithmetic unit 102 constitutes a control unit 201, a detection unit 202, a tracking arithmetic unit 203, a focus arithmetic unit 204, and an exposure arithmetic unit 205. The control unit 201 (including the exposure control unit, focus control unit, depth control unit, and control means determination unit of the present case) controls each part of the camera body 101 and the lens unit 120.

[0015] The detection unit 202 is composed of three components: a detector 213, a target area determination unit 214, and a priority area determination unit 215. The detector 213 performs the process of detecting a specific area (for example, a human face or eyes, an animal face or eyes) from an image. The specific area may not be detected or may be detected in multiple numbers. The detector for human or animal eyes is included in the detector 213. As the detection method, any known method such as AdaBoost or a convolutional neural network may be used. Also, as the implementation form, it may be a program running on a CPU, may be dedicated hardware, or may be a combination of them.

[0016] The subject detection result obtained from the detector 213 is sent to the target area determination unit 214, and one or a plurality of subjects such as the detected person and subject parts such as eyes are selected and determined as the target area to be used for the depth-first control described later. The determination of the target area is performed using a known calculation method based on the detected subject, the type, size, position of the subject part, and the reliability of the detection result. In addition to the subjects such as the person detected by the detector 213 and the subject parts such as eyes, it is also possible to determine the target area based on features such as past detection results and the edges of the target frame, and the defocus information of the subject (also referred to as information related to the subject distance).

[0017] The priority area determination unit 215 determines the priority order for each of the target areas determined by the target area determination unit 214. Regarding the priority order, it is also possible to determine only the target area with the highest priority, or to assign a priority to each of the target areas.

[0018] The tracking calculation unit 203 performs the tracking process of the target area based on the detection information of the target area. The tracking method may use a known method such as template matching that compares feature amounts between frames.

[0019] The focus calculation unit 204 calculates the defocus information for focusing.

[0020] The exposure calculation unit 205 calculates control values ​​for the aperture 122, the ISO sensitivity of the image sensor 104, and the shutter speed 103 to achieve proper exposure in the main subject area. Here are some specific examples of the calculation of control values. When the aperture 122 is controlled to a smaller value, the unit calculates a control value to reduce the amount of amplification (gain) of the signal used to generate the captured image obtained by the image sensor 104 in order to properly control the exposure. It also calculates a control value to shorten the time the shutter speed 103 is open (increase the shutter speed). When the aperture 122 is controlled to a larger value, the unit calculates a control value to increase the gain in order to achieve proper exposure. It also calculates a control value to decrease the shutter speed.

[0021] The depth information calculation unit 206 acquires defocus information and calculates positional information in the depth direction from the camera to the subject as depth information. Based on the calculated positional information, it also calculates the difference in the depth direction positions of multiple subjects as depth difference information. In this embodiment, depth information was calculated using the defocus amount calculated by the phase difference detection method, but it is also possible to acquire depth information using a depth sensor such as a LiDAR sensor that acquires depth information using the reflection of laser light. Any known method can be used to acquire depth information.

[0022] The control unit 201 receives the results from the detection unit 202, exposure calculation unit 205, and focus calculation unit 204, and controls the focus lens 121, aperture 122, display unit 105, etc. The control unit 201 also includes a depth control unit 216. When multiple target areas are set by the detection unit 202, the depth control unit 216 uses the depth information from the depth calculation unit 206 to determine whether it is possible to fit the multiple target areas within a specified depth, and if so, calculates the control values ​​for the lens and aperture. Based on the calculated control values, it controls the aperture 122. In addition, based on the control results, the display unit 105 displays a frame on the display screen indicating whether the subject is in focus or out of focus. Here, the specified depth generally refers to the depth of field, but it may also be an arbitrarily set depth. A subject that is within the specified depth (depth of field) is defined as being in focus.

[0023] Furthermore, the control unit 201 includes a focus control unit 217. The focus control unit 217 uses the defocus information from the focus calculation unit 204 and the depth information from the depth information calculation unit 206 to calculate a focus control value and control the focus lens 121.

[0024] The control unit 201 includes a control method determination unit 218. The control method determination unit 218 determines the control method for the depth control unit 216 and the focus control unit 217.

[0025] The control unit 106 is equipped with a release switch and a mode dial, and the control unit 201 can receive shooting instructions and mode change instructions from the user through the control unit 106.

[0026] Next, the pixel arrangement of the image sensor 104 will be explained using Figure 3. Figure 3 shows the pixel arrangement of the pixels (imaging pixels) that make up the image sensor 104, in a 4x4 range, as viewed from the optical axis direction (z direction).

[0027] Each pixel group 300 contains four imaging pixels arranged in a 2x2 grid. By arranging a large number of pixel groups 300 on the image sensor 104, photoelectric conversion of a two-dimensional subject image can be performed. Of each pixel group 300, the upper left contains an imaging pixel 300R with spectral sensitivity for red (R) (hereinafter referred to as "R pixel"), the upper right and lower left contain imaging pixels 300G with spectral sensitivity for green (G) (hereinafter referred to as "G pixel"). Furthermore, the lower right contains an imaging pixel 300B with spectral sensitivity for blue (B) (hereinafter referred to as "B pixel"). Each imaging pixel also contains a first focus detection pixel 301 and a second focus detection pixel 302, which are divided in the horizontal direction (x direction).

[0028] In this embodiment, the case in which each imaging pixel is divided into two horizontally is described, but it may also be divided vertically. Furthermore, although the image sensor 104 in this embodiment has a plurality of imaging pixels, each including a first and a second focus detection pixel, the imaging pixels and the first and second focus detection pixels may be provided as separate pixels. For example, the first and second focus detection pixels may be discretely arranged within a plurality of imaging pixels.

[0029] Figure 4(a) shows one imaging pixel 300 (300R, 300G, 300B) as viewed from the light-receiving side (+z direction) of the image sensor 104. Figure 4(b) shows a cross-sectional view of the aa cross section of the imaging pixel in Figure 4(a) as viewed from the -y direction. As shown in Figure 4(b), one imaging pixel is provided with one microlens 305 for focusing incident light.

[0030] Furthermore, the imaging pixel is provided with photoelectric conversion units 301 and 302, which are divided into N sections (2 sections in this embodiment) in the x direction. The photoelectric conversion units 301 and 302 correspond to the first focus detection pixel 301 and the second focus detection pixel 302, respectively. The centers of gravity of the photoelectric conversion units 301 and 302 are eccentric to the -x side and the +x side, respectively, with respect to the optical axis of the microlens 305.

[0031] A color filter 306 of R, G, or B is provided between the microlens 305 and the photoelectric conversion units 301 and 302 in each imaging pixel. The spectral transmittance of the color filter may be changed for each photoelectric conversion unit, or the color filter may be omitted.

[0032] Light incident on the imaging pixel via the imaging optical system is focused by the microlens 305, dispersed by the color filter 306, and then received by the photoelectric conversion units 301 and 302, where it is converted into electricity.

[0033] In each pixel with this configuration, the signal obtained by adding the signals from the photoelectric conversion units 301 and 302 (A+B signal) is used as the imaging signal, and the two signals (A signal and B signal) read out from each of the individual photoelectric conversion units 301 and 302 are used as a pair of focus detection signals. Although the imaging signal and focus detection signals may be read out separately, considering the processing load, the following method may also be used: That is, the imaging signal (A+B signal) and the focus detection signal from either the photoelectric conversion unit 301 or 302 (e.g., A signal) are read out, and the difference is taken to obtain the other focus detection signal with parallax (e.g., B signal). Alternatively, the focus detection signals (A signal and B signal) may be read out separately and added together to obtain the imaging signal (A+B signal).

[0034] A camera 100 having an image sensor 104 composed of pixels as shown in Figures 3 and 4 can perform so-called phase-difference focus detection by detecting a phase difference from the signal sequence of the pair of focus detection signals described above, using known techniques (for example, Japanese Patent Application Publication No. 2023-95509). Phase-difference focus detection makes it possible to detect the amount of defocus in a predetermined area within the shooting range, including the direction of defocus.

[0035] Next, the focus detection region of the image sensor 104, which is the region that acquires a signal sequence of a pair of focus detection signals for detecting phase difference, will be explained using Figure 5. The shift regions 503 on both sides of the focus detection region 502, which is set in the effective pixel region 501 of the image sensor 104, are the regions necessary for correlation calculation. Therefore, the pixel region 504, which is the sum of the focus detection region 502 and the shift region 503, becomes the pixel region necessary for correlation calculation. In Figure 6, p, q, s, and t each represent coordinates in the horizontal direction (x-axis direction), with p and q being the x-coordinates of the start and end points of the pixel region 504, respectively, and s and t being the x-coordinates of the start and end points of the focus detection region 502, respectively.

[0036] <Explanation of focus detection process> Next, the focus detection process will be explained with reference to the flowchart in Figure 6.

[0037] In step S601, "Focus Detection Area Setting," the focus calculation unit 204 sets an arbitrary range of focus detection areas 502 from the two-dimensionally arranged focus detection areas 502 within the imaging screen (see Figure 5). Then, the process proceeds to step S602.

[0038] In step S602, "image data acquisition," the focus calculation unit 204 acquires a pair (two) image signals (image A, image B) for focus detection from the image sensor 104 for the set focus detection region 502.

[0039] In step S603, "vertical row averaging," the focus calculation unit 204 performs vertical row averaging on the acquired pair of image signals to reduce the influence of noise. Here, "vertical direction" refers to the direction of extension of the vertical signal line (vertical transmission path) of the image sensor 104. In the embodiment of the present invention, when high-speed calculation processing is desired, such as in continuous shooting mode, the number of vertical row averaging operations is reduced, and in scenes where signal noise is noticeable, such as in dark places, the number of vertical row averaging operations is increased. Then the process proceeds to step S604.

[0040] In step S604, "Subject contrast value calculation," the focus calculation unit 204 calculates the subject contrast value CNT defined by the following equation (1). CNT = (Peak - Bottom) / Peak (1)

[0041] Here, Peak is a variable that indicates the maximum value (maximum output value) of the waveform averaged vertically, and Bottom is a variable that indicates the minimum value (minimum output value) of the waveform averaged vertically. The focus calculation unit 204 calculates the subject contrast value CNT by dividing the difference between the maximum and minimum values ​​of the waveform averaged vertically by the maximum value, as shown in equation (1). The subject contrast value CNT is used to evaluate the reliability of the amount of defocusing.

[0042] In step S605, the "filtering process," the focus calculation unit 204 performs a filtering process to extract signal components of a predetermined frequency band from the signal that was vertically averaged in step S603. In this embodiment of the present invention, three types of filters (low-frequency band filter, medium-frequency band filter, and high-frequency band filter) with different extraction frequency bands are prepared in advance. The defocus amount to be used from among the defocus amounts calculated using each filter is switched according to the degree of blurring of the subject. Using the low-frequency band filter improves the distance measurement performance (defocus amount calculation performance) for heavily blurred subjects where the edges of the subject are distorted. Using the high-frequency band filter allows for highly accurate distance measurement near the point of focus where the edges of the subject are sharp (the accuracy of the defocus amount calculation can be improved). Note that the configuration is not limited to using three types of filters; any configuration using at least one type of filter is acceptable.

[0043] In S606, "Calculation of the correlation amount COR between image signals," the focus calculation unit 204 calculates the correlation amount COR between a pair (two) of acquired image signals (i.e., signal components of a predetermined frequency band extracted by filtering). In this embodiment of the present invention, this calculation is referred to as "correlation calculation." The focus calculation unit 204 performs the correlation calculation for each scan line after vertical averaging within the focus detection area.

[0044] In step S607, "Correlation Amount COR Addition," the focus calculation unit 204 adds the waveforms of the correlation amount COR within the focus detection area.

[0045] In step S608, "Calculation of Correlation Change," the focus calculation unit 204 calculates the correlation change from the correlation amount COR.

[0046] In step S609, "Calculation of maxder and image displacement," the focus calculation unit 204 calculates the displacement p of the two images (image A and image B) based on the calculated correlation change. Furthermore, the focus calculation unit 204 calculates the steepness of the correlation change (hereinafter referred to as maxder).

[0047] In step S610, "Defocus Amount Calculation Conversion," the focus calculation unit 204 calculates the defocus amount d by multiplying the amount of displacement p between the two images calculated in step S609 by a predetermined conversion coefficient k. The conversion coefficient k used in this process is determined by the aperture value, the exit pupil distance of the lens, the individual information of the image sensor 104, and the coordinates for setting the focus detection area 1102, and is stored in ROM (not shown) in advance. The focus calculation unit 204 then normalizes the calculated defocus amount d by dividing it by the aperture value and the allowable circle of confusion δ, making it possible to evaluate the amount of focus shift using the same index even if the aperture value is different.

[0048] In step S611, the "reliability evaluation," the focus calculation unit 204 evaluates the reliability of the defocus amount d calculated in step S610 based on the maxder (steepness) calculated in step S609. Details of the reliability evaluation process in step S1011 will be described later.

[0049] In step S612, "Have calculations been performed for each type of filter?", the focus calculation unit 204 determines whether the processes in steps S605 to S611 have been performed for all three types of filters that have been prepared in advance. Here, the three types of filters are a low-frequency band filter, a medium-frequency band filter, and a high-frequency band filter. If there are still filters that have not been executed ("No"), the process returns to step S605 and performs the processes in steps S605 to S611 for the filters that have not yet been executed. If the processes have been performed for all types of filters ("Yes"), this focus detection process is terminated.

[0050] <Explanation of depth control> Next, referring to Figure 7, we will explain a method of controlling depth of field by controlling the aperture so that multiple detected subjects or parts of multiple subjects are at the same depth of field. Figure 7 shows the state in which the main subject 701 and the secondary subject 702 have been detected. First, the amount of defocus is obtained as depth information for each subject. In the example in Figure 7, the amount of defocus Def1 for subject 701 and the amount of defocus Def2 for subject 702 are obtained.

[0051] The difference in the amount of defocus is taken as the depth of field difference (DefRange) between subjects, and an aperture value F is calculated such that the depth of field difference falls within a predetermined range. For example, the predetermined depth of field difference is set to ±Fδ, which is the product of the aperture value F and the allowable circle of confusion diameter δ. In this case, an aperture value F can be calculated to keep the main / subject within a range of ±1Fδ so as to satisfy equation (2). DefRange=|Def1-Def2|=1Fδ (2)

[0052] In this embodiment, the case with two subjects was described as an example, but the number of subjects is not limited. For three or more subjects, for example, the aperture value F may be determined so that the subjects with the maximum and minimum defocus amounts fall within the same depth of field.

[0053] <Explanation of focus position control> Furthermore, with reference to Figure 7, focus position control will be explained. In order to capture multiple detected subjects or parts of multiple subjects within the same depth of field with minimal aperture control, it is desirable to control the focus position to the center of the depth difference, which is the focal center position between each subject.

[0054] Therefore, the defocus amount Def1 of subject 701 and the defocus amount Def2 of subject 702 are obtained, and the defocus amount DefCenter, which controls the focus position to the center of focus (center of depth difference), is calculated using equation (3). DefCenter = (Def1 + Def2) / 2 (3)

[0055] In this embodiment, the case with two subjects was described as an example, but the number of subjects is not limited. For three or more subjects, the focus position may be controlled to the focal center position of the subject where the amount of defocus is maximum or minimum.

[0056] <Problems caused by vibrational changes in aperture value or focus position> When trying to capture multiple moving subjects within the depth of field, each subject does not necessarily move in the same direction as the subject in focus. Therefore, the depth of field difference between the multiple subjects, and the center of focus, tend to change oscillatingly during the shoot.

[0057] When the time-dependent change in the depth of field between multiple subjects is oscillating, alternating between deep and shallow, the aperture value required to keep all subjects within the depth of field will also change oscillatingly. This oscillating change in aperture value can cause flickering of brightness, raising concerns about a decrease in the quality of still images and videos.

[0058] Furthermore, if the time-dependent change in the focal point position of multiple subjects is oscillating, alternating between the near and infinity ends, continuously focusing on the focal point will cause the focus position to change oscillatingly. When the focus position changes oscillatingly, the degree of blur in the background and near objects will fluctuate, raising concerns about a decrease in the quality of still images and videos, especially with bright aperture values.

[0059] Therefore, in order to suppress the deterioration of still images and videos due to the above vibration conditions, the following will be taken in this case. (1) Switching of depth control method by detecting the vibration state of the depth difference. (2) Switching of focus position control method by detecting the vibration state of the focus position <Detection of vibration state> The method for detecting the vibration state of the depth difference and focus position will be explained with reference to Figures 8-12.

[0060] First, the time change in the amount of defocus of the detected multiple subjects is calculated. Figure 8 shows the state in which the main subject 801, secondary subject 802, and secondary subject 803 are detected. Figure 9 shows an image diagram of the time change in the amount of defocus of each detected subject. In Figure 9, defocus 901 corresponds to the amount of defocus of the main subject 801, defocus 902 corresponds to the amount of defocus of the secondary subject 802, and defocus 903 corresponds to the amount of defocus of the secondary subject 803. Here, the amount of defocus at T=t+Δt, after a certain time Δt has elapsed from the current time T=t, is predicted from the time history information of past defocus amounts as shown in equation (4), and the prediction result is obtained. Def(t+Δt)=Def(t)+(Def(t)-Def(t-Δt)) (4)

[0061] Next, the time evolution of the depth of field and focus position between each subject is calculated from the amount of defocus for each subject. Figure 10 shows an image of the time evolution of the depth of field and focus position between each subject. In Figure 10, 1001 corresponds to the depth of field and 1002 corresponds to the focus position. Here, the depth of field DefRange(t+Δt) and the focus position DefCenter(t+Δt) corresponding to the center of focus position for each subject at time T=t+Δt are calculated as shown in equations (5) and (6). DefRange(t+Δt)=max(Def1(t+Δt),Def2(t+Δt),Def3(t+Δt))-min(Def1(t+Δt),Def2(t+Δt),Def3(t+Δt)) (5) DefCenTer(t+Δt)=(max(Def1(t+Δt),Def2(t+Δt),Def3(t+Δt))+min(Def1(t+Δt),Def2(t+Δt),Def3(t+Δt)) / 2 (6)

[0062] Next, we calculate the acceleration (rate of change) of the depth of field and the focus position. Figures 11(a) and (b) show an image of the time change of the depth of field and the acceleration of the focus position for each subject. In Figures 11(a) and (b), 1101 corresponds to the acceleration of the depth of field, 1102 to the acceleration of the focus position, 1103 to the vibration detection period, and 1104 to the dead zone. Here, the acceleration is calculated by taking the second derivative with respect to the depth of field DefRange and the focus position DefCenter shown in equations (5) and (6).

[0063] Next, the vibration state is determined. In this embodiment, the number of sign reversals in the time change of the depth difference or the acceleration of the focus position is counted, and if the number of sign reversals within a predetermined vibration determination period is greater than or equal to a predetermined value, it is determined that a vibration state exists. However, simply counting the number of sign reversals will also count sign reversals caused by noise in the acceleration change, leading to an overestimation of the vibration state. Therefore, it is preferable to implement noise countermeasures by providing a dead zone.

[0064] The method for determining the vibration state described above will be explained in more detail with reference to Figures 11(a) and 11(b).

[0065] Figure 11(a) shows an example of the time change of acceleration when it is determined that the system is in a state of vibration, and Figure 11(b) shows an example of the time change of acceleration when it is determined that the system is not in a state of vibration.

[0066] In the example shown in Figure 11(a), if the number of times the dead zone 1104 is exceeded and the sign is reversed during the vibration determination period 1103 is greater than or equal to a predetermined number (for example, 5 times or more), then both the acceleration of the depth difference and the acceleration of the focus position are determined to be in a vibration state.

[0067] In the example shown in Figure 11(b), during the vibration detection period 1103, the acceleration does not exceed the dead zone 1104, and both the depth difference acceleration 1101 and the focus position acceleration 1102 are determined to be in a non-vibrating state. Thus, if vibration detection is performed using only the sign of acceleration, the influence of noise may cause excessive sign reversal to be detected. Therefore, by introducing a dead zone for sign detection as described above, it becomes possible to detect sign reversal with high accuracy.

[0068] Furthermore, in addition to setting a dead zone, a low-pass filter is applied to the time-dependent changes in the depth of field or the acceleration of the focus position to suppress noise. In this case, since the frequency band for noise suppression differs depending on whether the frequency of the time-dependent change is large or small depending on the movement of the subject, it is preferable to apply a filter with responsiveness corresponding to the frequency of the time-dependent change. When the frequency is large, a filter with a short tap length and high responsiveness is applied, and when the frequency is small, a filter with a long tap length and low responsiveness is applied. Alternatively, the filter may be applied not to the time-dependent changes in the depth of field or the acceleration of the focus position, but to the time-dependent changes in the amount of defocus or the depth of field or focus position. This improves the robustness of the vibration state determination accuracy with respect to the movement of the subject.

[0069] <Switching aperture control method> If the depth of field is not fluctuating, it is preferable to control the aperture to track the depth of field changes to ensure a sufficient depth of field, thereby preventing over-stopping down. The reason why over-stopping down is necessary is that stopping down the aperture makes the image darker, and in order to maintain proper exposure, it becomes necessary to increase the ISO sensitivity or extend the exposure recovery time. Increasing the ISO sensitivity increases the amount of noise, leading to a decrease in the quality of still images and videos. Also, extending the exposure recovery time makes motion blur more likely in moving subjects.

[0070] Conversely, if the depth of field is fluctuating, controlling the aperture to follow the depth of field changes will cause the brightness to flicker, resulting in a degradation of still image and video quality that is even greater than the degradation caused by excessive stopping down. Therefore, when the depth of field is fluctuating, it is preferable to control the aperture without following the depth of field changes. However, it is preferable to be able to change the aperture only in the darker direction so that the group of subjects can be framed at the same depth of field.

[0071] Therefore, the imaging device in this embodiment has the following two depth control methods. (1) First depth control method: Allows the aperture value to be changed to either the brighter or darker direction. (2) Second depth control method: Allows the aperture value to be changed only in the dark direction. When the time-dependent change in depth of field is oscillating, selecting a second depth control method can suppress aperture vibration.

[0072] <Switching focus position control method> If the focus position is not vibrating, it is preferable to set the focus position to the center of focus of the subject group and ensure that the entire subject group is in focus with the minimum necessary aperture amount, thereby preventing over-stopping down.

[0073] Conversely, if the focus position is fluctuating, tracking the focus position to the center of focus of the subject group will cause the shooting angle and bokeh to change erratically, resulting in a degradation of still image and video quality that is even greater than the degradation caused by stopping down the aperture too much. Therefore, when the focus position is fluctuating, it is preferable to set the focus position so that the main subject is in focus.

[0074] The imaging device in this embodiment has the following two methods for controlling the focus position. (1) First focus control method: Controls the focus position to the center position of focus between multiple subjects. (2) Second focus control method: Controls the focus position so that the main subject is in focus among multiple subjects. When the time-dependent change in the focus position is oscillating, selecting a second focus control method can suppress the oscillation of the focus position.

[0075] <Explanation of the switching process between depth control and focus control methods> Using Figure 12, the flow for switching between the depth control method and the focus control method in this embodiment will be explained.

[0076] In step S1201, the main / subject is detected from the image. In step S1202, the amount of defocus of the main / subject after a predetermined time has elapsed, such as the next frame, is predicted based on the time history information of the defocus amount.

[0077] In the following steps S1203 to S1209, the depth control method is switched.

[0078] First, in step S1203, the depth difference between the main and secondary subjects is calculated based on the predicted amount of defocus. In addition, time history information of the depth difference up to the current frame is also retained.

[0079] Next, in step S1204, the aperture value Fobj is calculated to bring the main and secondary subjects to the same depth of field in the next frame. In step S1205, the acceleration of the depth difference is calculated using the depth difference of the main and secondary subjects obtained in step S1203. In addition, the time history information of the acceleration of the depth difference up to the current frame is also stored.

[0080] In step S1206, the time history information of the acceleration of the depth difference within the depth determination time is referenced to determine the oscillation state of the time change of the depth difference. At this time, a filter with a low-pass effect is applied to the time history information of the acceleration of the depth difference. Alternatively, multiple statistical processes are used.

[0081] If, within the vibration determination time, the acceleration of the depth difference does not exceed the dead zone, or the number of sign reversals is less than a predetermined number, it is determined that the depth difference is not in a vibrating state, and the process proceeds to step S1207.

[0082] If, within the vibration determination time, the acceleration of the depth difference exceeds the dead zone and the number of sign reversals is greater than or equal to a predetermined number, it is determined that the depth difference is in a vibration state, and the process proceeds to step S1208.

[0083] In step S1207, set the aperture value to aperture value Fobj.

[0084] In step S1208, if the aperture value Fobj is darker than the aperture value Fprev of the current frame, proceed to step S1207. If the aperture value Fobj is brighter than the aperture value Fprev of the current frame, proceed to step S1209.

[0085] In step S1209, set the aperture value to Fprev.

[0086] In the following steps S1210 to S1214, the focus control method is switched.

[0087] First, in step S1210, the focus position, which will be the center of focus for the main / subject, is calculated based on the predicted amount of defocus. In addition, time history information of the focus position up to the current frame is also retained.

[0088] Next, in step S1211, the acceleration of the focus position is calculated using the focus position obtained in step S1210. In addition, the time history information of the acceleration of the focus position up to the current frame is also stored.

[0089] In step S1212, the vibration state of the focus position is determined by referring to the time history information of the acceleration of the focus position within the vibration determination time. At this time, a filter with a low-pass effect is applied to the time history information of the acceleration of the focus position. Alternatively, multiple statistical processes are used.

[0090] If, within the vibration detection time, the acceleration of the focus position does not exceed the dead zone, or if the sign reversal occurs less than a predetermined number of times, it is determined that the focus position is not in a vibrating state, and the process proceeds to step S1213.

[0091] If, within the vibration determination time, the acceleration of the focus position exceeds the dead zone and the sign reversal occurs more than a predetermined number of times, it is determined that the focus position is in a vibration state, and the process proceeds to step S1214.

[0092] In step S1213, the focus position is set to the center of focus of the main / secondary subject.

[0093] In step S1214, the focus position is set to the focal point of the main subject.

[0094] In this embodiment, both depth control and focus control methods are switched, but it is also possible to determine whether to switch only one of the control methods.

[0095] The above describes a method for detecting vibrations in at least one of the depth difference or focus position and automatically switching between at least one of the depth control method or focus control method.

[0096] Alternatively, the camera may be configured to allow the user to directly specify a shooting mode from the camera's menu screen, and to select a control method according to the specified shooting mode. Figure 13 shows an example of the camera's menu screen for specifying a shooting mode. In Figure 13, 1301 is the button display for switching the shooting mode "Smooth Depth of Field Priority," which corresponds to switching the depth of field control method, and 1302 is the button display for switching the shooting mode "Smooth Focus Position Priority," which corresponds to switching the focus control method. When button display 1301 is OFF, the aforementioned first depth of field control method (allowing the aperture value to be changed in either the bright or dark direction) is selected. On the other hand, when button display 1301 is ON, the aforementioned second depth of field control method (allowing the aperture value to be changed only in the dark direction) is selected. Also, when button display 1302 is OFF, the aforementioned first focus control method (changing the focus position to the center of focus between multiple subjects) is selected. On the other hand, when button display 1302 is ON, the second depth control method described above (changing the focus position so that the main subject is in focus among multiple subjects) is selected. Furthermore, switching ON / OFF is not limited to touch operation of the switch, but can also be done using voice input or eye-tracking input. In this way, by allowing the user to directly specify the shooting mode and selecting the depth control method or focus control method linked to the specified shooting mode, depth control and focus control that match the user's intentions become possible.

[0097] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0098] <Summary> The above embodiments include the following configuration.

[0099] (Composition 1) A calculation means for calculating depth difference information between multiple subjects, It includes depth control means that controls the aperture based on the depth difference information, The depth control means is characterized by switching the aperture control method depending on whether the time change of the depth difference information is in a oscillating state or not.

[0100] (Configuration 2) A means for acquiring positional information in the depth direction of multiple subjects, It includes a focus control means that controls the focus position based on the position information in the depth direction, The imaging apparatus is characterized in that the focus control means switches the focus position control method depending on whether the time change of the focal center position based on the depth position information is in a vibrating state or not.

[0101] (Composition 3) A calculation means for calculating depth difference information between multiple subjects, A depth control means that controls the aperture based on the depth difference information, A means for acquiring positional information in the depth direction of multiple subjects, It includes a focus control means that controls the focus position based on the position information in the depth direction, The depth control means switches the aperture control method depending on whether the time change of the depth difference information is in a oscillating state or not. The imaging apparatus is characterized in that the focus control means switches the focus position control method depending on whether the time change of the focal center position based on the depth position information is in a vibrating state or not.

[0102] (Composition 4) The imaging apparatus according to configuration 1 or 2, characterized in that the depth control means includes a first depth control method that allows the aperture value to be changed in either the brighter or darker direction, and a second depth control method that allows the aperture value to be changed only in the darker direction.

[0103] (Composition 5) The imaging apparatus according to configuration 2 or 3, characterized in that the focus control means includes, as a method for controlling the focus position, a first focus control method for controlling the focus position to the center of focus between the multiple subjects, and a second focus control method for controlling the focus position so that any of the multiple subjects is in focus.

[0104] (Composition 6) The imaging apparatus according to configuration 4, characterized in that the depth control means selects the second depth control method when the time change of the depth difference information is in an oscillating state.

[0105] (Composition 7) The imaging apparatus according to configuration 5, characterized in that the focus control means selects the second focus control method when the time change of the focal center position based on the position information in the depth direction is in an oscillating state.

[0106] (Composition 8) The imaging apparatus according to configuration 6, characterized in that the case where the time change of the depth difference information is in an oscillating state is a state in which the change in aperture value when the aperture is controlled in accordance with the change in depth changes alternately in the brighter direction and the darker direction.

[0107] (Composition 9) The imaging apparatus according to configuration 7, characterized in that the time change of the focal center position based on the depth direction position information is in an oscillating state, which means that the focal center position based on the depth direction position information alternately changes between the near side and the infinity side.

[0108] (Composition 10) The imaging apparatus according to configuration 6, characterized in that the time change of the depth difference information is in a oscillating state, in which the sign of the acceleration of the depth difference information changes a predetermined number of times or more in a predetermined time.

[0109] (Composition 11) The imaging apparatus according to configuration 7, characterized in that the time change of the focal center position based on the depth direction position information is in a state of oscillation, where the sign of the acceleration of the focal center position information based on the depth direction position information changes a predetermined number of times or more in a predetermined time.

[0110] (Composition 12) The imaging apparatus according to configuration 10, characterized in that the acceleration of the depth difference information is calculated using multiple filter processing with a low-pass effect, or multiple statistical processing.

[0111] (Composition 13) The imaging apparatus according to configuration 11, characterized in that the acceleration of the information of the focal center position based on the position information in the depth direction is calculated using multiple filter processing with a low-pass effect or multiple statistical processing.

[0112] (Composition 14) The imaging apparatus according to any one of configurations 1 to 3, further comprising a detection means for detecting the multiple subjects from the image.

[0113] (Composition 15) The imaging apparatus according to any one of configurations 1 to 3, characterized in that the depth control means controls the aperture so that the multiple subjects are within the depth of field.

[0114] (Composition 16) The imaging apparatus according to configuration 1, further comprising prediction means for acquiring prediction results that predict the positions in the depth direction of the plurality of subjects after a predetermined time has elapsed.

[0115] (Composition 17) The imaging apparatus according to configuration 16, wherein the depth control means changes the aperture based on the prediction result.

[0116] (Composition 18) The imaging apparatus according to configuration 2, further comprising prediction means for acquiring prediction results that predict the positions in the depth direction of the plurality of subjects after a predetermined time has elapsed.

[0117] (Composition 19) The imaging apparatus according to configuration 18, characterized in that the focus control means controls the focus position based on the prediction result.

[0118] (Composition 20) The imaging device according to configuration 1, characterized in that the depth control means switches the control method according to the shooting mode set by the user.

[0119] (Composition 21) The imaging device according to configuration 2, characterized in that the focus control means switches the control method according to the shooting mode set by the user.

[0120] (Method 1) A calculation step to calculate depth difference information between multiple subjects, The depth control step includes controlling the aperture based on the depth difference information, The control method for an imaging device is characterized in that, in the depth control step, the aperture control method is switched depending on whether the time change of the depth difference information is in a oscillating state or not.

[0121] (Method 2) An acquisition step to obtain positional information in the depth direction of multiple subjects, The system includes a focus control step that controls the focus position based on the position information in the depth direction, The control method for an imaging device is characterized in that, in the focus control step, the control method for the focus position is switched depending on whether or not the time change of the center of focus position based on the position information in the depth direction is in a vibrating state.

[0122] (Method 3) A calculation step to calculate depth difference information between multiple subjects, A depth control step that controls the aperture based on the depth difference information, An acquisition step to obtain positional information in the depth direction of multiple subjects, The system includes a focus control step that controls the focus position based on the position information in the depth direction, In the depth control step, the aperture control method is switched depending on whether the time change of the depth difference information is in a oscillating state or not. The control method for an imaging device is characterized in that, in the focus control step, the control method for the focus position is switched depending on whether or not the time change of the center of focus position based on the position information in the depth direction is in a vibrating state.

[0123] (Program 1) A program for causing a computer to function as one of the means of an imaging apparatus described in any one of items 1 to 3 of the configuration.

[0124] (Storage medium 1) A computer-readable storage medium containing a program for causing the computer to function as one of the means of the imaging device described in any one of items 1 to 3.

Claims

1. A calculation means for calculating depth difference information between multiple subjects, It includes depth control means that controls the aperture based on the depth difference information, The depth control means is characterized by switching the aperture control method depending on whether the time change of the depth difference information is in a oscillating state or not.

2. A means for acquiring positional information in the depth direction of multiple subjects, It includes a focus control means that controls the focus position based on the position information in the depth direction, The imaging apparatus is characterized in that the focus control means switches the focus position control method depending on whether the time change of the focal center position based on the depth position information is in a vibrating state or not.

3. A calculation means for calculating depth difference information between multiple subjects, A depth control means that controls the aperture based on the depth difference information, A means for acquiring positional information in the depth direction of multiple subjects, It includes a focus control means that controls the focus position based on the position information in the depth direction, The depth control means switches the aperture control method depending on whether the time change of the depth difference information is in a oscillating state or not. The imaging apparatus is characterized in that the focus control means switches the focus position control method depending on whether the time change of the focal center position based on the depth position information is in a vibrating state or not.

4. The imaging apparatus according to claim 1 or 2, characterized in that the depth control means includes a first depth control method that allows the aperture value to be changed in either the brighter or darker direction, and a second depth control method that allows the aperture value to be changed only in the darker direction.

5. The imaging apparatus according to claim 2 or 3, characterized in that the focus control means includes, as a method for controlling the focus position, a first focus control method for controlling the focus position to the center of focus between the multiple subjects, and a second focus control method for controlling the focus position so that any of the multiple subjects is in focus.

6. The imaging apparatus according to claim 4, characterized in that the depth control means selects the second depth control method when the time change of the depth difference information is in an oscillating state.

7. The imaging apparatus according to claim 5, characterized in that the focus control means selects the second focus control method when the time change of the focal center position based on the position information in the depth direction is in an oscillating state.

8. The imaging device according to claim 6, characterized in that the case where the time change of the depth difference information is in an oscillating state is a state in which the change in aperture value when the aperture is controlled in accordance with the change in depth changes alternately in the brighter direction and the darker direction.

9. The imaging apparatus according to claim 7, characterized in that the time change of the focal center position based on the depth direction position information is in an oscillating state, which means that the focal center position based on the depth direction position information alternately changes between the near side and the infinity side.

10. The imaging apparatus according to claim 6, characterized in that the time change of the depth difference information is in a state of vibration, in which the sign of the acceleration of the depth difference information changes a predetermined number of times or more in a predetermined time.

11. The imaging apparatus according to claim 7, characterized in that the time change of the focal center position based on the depth position information is in a state of vibration, where the sign of the acceleration of the focal center position information based on the depth position information changes a predetermined number of times or more in a predetermined time.

12. The imaging apparatus according to claim 10, characterized in that the acceleration of the depth difference information is calculated using multiple filtering processes with a low-pass effect, or multiple statistical processing processes.

13. The imaging apparatus according to claim 11, characterized in that the acceleration of the information of the focal center position based on the position information in the depth direction is calculated using multiple filter processing with a low-pass effect or multiple statistical processing.

14. The imaging device according to any one of claims 1 to 3, further comprising a detection means for detecting the multiple subjects from the image.

15. The imaging apparatus according to any one of claims 1 to 3, characterized in that the depth control means controls the aperture so that the multiple subjects are within the depth of field.

16. The imaging apparatus according to claim 1, further comprising prediction means for acquiring prediction results that predict the positions in the depth direction of the plurality of subjects after a predetermined time has elapsed.

17. The imaging apparatus according to claim 16, characterized in that the depth control means changes the aperture based on the prediction result.

18. The imaging apparatus according to claim 2, further comprising prediction means for acquiring prediction results that predict the positions in the depth direction of the plurality of subjects after a predetermined time has elapsed.

19. The imaging apparatus according to claim 18, characterized in that the focus control means controls the focus position based on the prediction result.

20. The imaging device according to claim 1, characterized in that the depth control means switches the control method according to the shooting mode set by the user.

21. The imaging device according to claim 2, characterized in that the focus control means switches the control method according to the shooting mode set by the user.

22. A calculation step to calculate depth difference information between multiple subjects, The depth control step includes controlling the aperture based on the depth difference information, The control method for an imaging device is characterized in that, in the depth control step, the aperture control method is switched depending on whether the time change of the depth difference information is in a oscillating state or not.

23. An acquisition step to obtain positional information in the depth direction of multiple subjects, The system includes a focus control step that controls the focus position based on the position information in the depth direction, The control method for an imaging device is characterized in that, in the focus control step, the control method for the focus position is switched depending on whether or not the time change of the center of focus position based on the position information in the depth direction is in a vibrating state.

24. A calculation step to calculate depth difference information between multiple subjects, A depth control step that controls the aperture based on the depth difference information, An acquisition step to obtain positional information in the depth direction of multiple subjects, The system includes a focus control step that controls the focus position based on the position information in the depth direction, In the depth control step, the aperture control method is switched depending on whether the time change of the depth difference information is in a oscillating state or not. The control method for an imaging device is characterized in that, in the focus control step, the control method for the focus position is switched depending on whether or not the time change of the center of focus position based on the position information in the depth direction is in a vibrating state.

25. A program for causing a computer to function as one of the means of an imaging apparatus described in any one of claims 1 to 3.

26. A computer-readable storage medium storing a program for causing a computer to function as one of the means of an imaging apparatus described in any one of claims 1 to 3.

Images