Control device, imaging device, control method, and program

Abstract

To provide a control device that enables stable focusing. [Solution] The control device (125) includes a first acquisition means (1251) that acquires first information relating to the first image plane position of the subject using a time-series focus detection result based on an image signal obtained from an image sensor (122), a second acquisition means (1252) that acquires second information relating to the future second image plane position of the subject using the first information, and a control means (1253) that controls the focus lens (104) using the first information or the second information according to the movement of the subject.

Description

[Technical Field]

[0001] The present invention relates to a control device, an imaging device, a control method, and a program. [Background technology]

[0002] Conventionally, imaging devices with an autofocus (AF) function that automatically adjusts focus based on the defocus amount, which is a focus detection result obtained using a signal from an image sensor, are known. Patent Document 1 discloses a method for suppressing variability by using a defocus amount obtained by averaging the defocus amounts detected over time, or a defocus amount detected after averaging the focus detection signals captured over time. Patent Document 2 discloses a method for predicting the future focus position of a subject from the defocus amount over time when the subject is moving, and predicting the future focus position of a subject from a smaller defocus amount over time when the subject is not moving. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2019-91031 [Patent Document 2] Japanese Patent Publication No. 2021-9197 [Overview of the project] [Problems that the invention aims to solve]

[0004] The method disclosed in Patent Document 1 assumes a stationary subject and may not be able to accurately detect the amount of defocus for a moving subject. The method disclosed in Patent Document 2, even when using a small time series of defocus amounts, can cause errors in predicting the future focus position of the subject if the defocus amount is inconsistent, leading to unstable control of the focus lens.

[0005] Therefore, an object of the present invention is to provide a control device capable of stable focusing.

Means for Solving the Problems

[0006] A control device according to one aspect of the present invention includes: a first acquisition unit that acquires first information regarding a first image plane position of a subject using a time-series focus detection result based on an image signal obtained from an imaging device; a second acquisition unit that acquires second information regarding a future second image plane position of the subject using the first information; and a control unit that controls a focus lens using the first information or the second information according to the movement of the subject.

[0007] Other objects and features of the present invention will be described in the following embodiments.

Effects of the Invention

[0008] According to the present invention, it is possible to provide a control device capable of stable focusing.

Brief Description of the Drawings

[0009] [Figure 1] It is a block diagram of an imaging device in the present embodiment. [Figure 2] It is a schematic diagram of a pixel array in the present embodiment. [Figure 3] It is a schematic plan view and a schematic cross-sectional view of a pixel in the present embodiment. [Figure 4] It is a schematic explanatory diagram of a pixel and pupil division in the present embodiment. [Figure 5] It is a schematic explanatory diagram of an imaging device and pupil division in the present embodiment. [Figure 6] It is a schematic relationship diagram of defocus amount and image shift amount in the present embodiment. [Figure 7] It is an explanatory diagram of Kalman filter calculation in the present embodiment. [Figure 8] It is a flowchart showing the movement determination process in the present embodiment. [Figure 9] This is a diagram showing an example of the image plane position of a subject used for movement determination in the present embodiment. [Figure 10] This is a diagram showing an example of the image plane position of a subject used for calculation of predicted image plane information in the present embodiment. [Figure 11] This is a flowchart showing the focus adjustment process in the present embodiment.

Embodiments for Carrying out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] (Configuration of Imaging Device) First, referring to FIG. 1, the configuration of the imaging device in the present embodiment will be described. FIG. 1 is a block diagram of an imaging system 10 (a single-lens reflex type digital camera system with interchangeable lenses) in the present embodiment. The imaging system 10 includes a lens unit (interchangeable lens, lens device) 100 and a camera body (imaging device) 120. The lens unit 100 is detachably attached to the camera body 120 via a mount M shown by a dotted line in FIG. 1. However, the present embodiment is not limited to this, and it is also applicable to an imaging device (digital camera) in which the lens unit (imaging optical system) and the camera body are integrally configured. Further, the present embodiment is not limited to a digital camera, and is also applicable to other imaging devices such as a video camera.

[0012] The lens unit 100 includes a first lens group 101 as an optical system, an aperture (aperture stop) 102, a second lens group 103, a focus lens (focus lens group) 104, and a drive / control system. Thus, the lens unit 100 includes the focus lens 104 and has an optical system (imaging optical system) for forming a subject image.

[0013] The first lens group 101 is positioned at the front of the lens unit 100 and is held so as to be able to move back and forth in the optical axis direction OA. The aperture 102 adjusts the amount of light during shooting by adjusting its aperture diameter, and also functions as a shutter for adjusting the exposure time in still image shooting. The aperture 102 and the second lens group 103 are able to move together in the optical axis direction OA, and a zoom function is realized in conjunction with the movement of the first lens group 101. The focus lens 104 is able to move in the optical axis direction OA, and the subject distance (focus distance) at which the lens unit 100 focuses changes depending on its position. By controlling the position of the focus lens 104 in the optical axis direction OA, it is possible to adjust the focus distance of the lens unit 100 (focus control).

[0014] The drive / control system includes a zoom actuator 111, an aperture actuator 112, a focus actuator 113, a zoom drive circuit 114, an aperture drive circuit 115, a focus drive circuit 116, a lens MPU 117, and a lens memory 118. The zoom drive circuit 114 uses the zoom actuator 111 to drive the first lens group 101 and the second lens group 103 in the optical axis direction OA, controlling the angle of view of the optical system of the lens unit 100 (performing zoom operation). The aperture drive circuit 115 uses the aperture actuator 112 to drive the aperture 102, controlling the aperture diameter and opening / closing operation of the aperture 102. The focus drive circuit 116 uses the focus actuator 113 to drive the focus lens 104 in the optical axis direction OA, controlling the focusing distance of the optical system of the lens unit 100 (performing focus control). The focus drive circuit 116 also functions as a position detection unit that uses the focus actuator 113 to detect the current position (lens position) of the focus lens 104.

[0015] The lens MPU (processor) 117 performs all calculations and controls related to the lens unit 100, and controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116. The lens MPU 117 is also connected to the camera MPU 125 via the mount M and communicates commands and data. For example, the lens MPU 117 detects the position of the focus lens 104 and notifies the camera MPU 125 of the lens position information in response to a request. This lens position information includes information such as the position of the focus lens 104 in the optical axis direction OA, the position and diameter of the exit pupil in the optical axis direction OA when the optical system is not moving, and the position and diameter of the lens frame in the optical axis direction OA that limits the light beam of the exit pupil. The lens MPU 117 also controls the zoom drive circuit 114, aperture drive circuit 115, and focus drive circuit 116 in response to a request from the camera MPU 125. The lens memory 118 stores the optical information necessary for autofocus (AF control). The camera MPU 125 controls the operation of the lens unit 100 by executing programs stored, for example, in the built-in non-volatile memory or lens memory 118.

[0016] The camera body 120 includes an optical low-pass filter 121, an image sensor 122, and a drive / control system. The optical low-pass filter 121 and the image sensor 122 function as an imaging unit (imaging means) that photoelectrically converts the subject image (optical image) formed via the lens unit 100 and outputs image data. In this embodiment, the image sensor 122 photoelectrically converts the subject image formed via the imaging optical system and outputs an imaging signal and a focus detection signal as image data, respectively. In this embodiment, the first lens group 101, aperture 102, second lens group 103, focus lens 104, and optical low-pass filter 121 constitute the imaging optical system.

[0017] The optical low-pass filter 121 reduces false colors and moiré patterns in captured images. The image sensor 122 consists of a CMOS image sensor and its peripheral circuits, with m pixels in the horizontal direction and n pixels in the vertical direction (m and n are integers of 2 or more). The image sensor 122 in this embodiment also functions as a focus detection element and has pupil division functionality, and has pupil division pixels that enable phase-difference detection (phase-difference AF) using image data (image signals). The image processing circuit 124 generates data for phase-difference AF and image data for display, recording, and subject recognition based on the image data output from the image sensor 122.

[0018] The drive / control system includes an image sensor drive circuit 123, an image processing circuit 124, a camera MPU 125, a display 126, a group of operation switches (operation SW) 127, and a memory 128. The drive / control system also includes a phase-detection AF unit (focus detection means) 129, an AE unit 130, a white balance adjustment unit 131, and a subject recognition unit 132.

[0019] The image sensor drive circuit 123 controls the operation of the image sensor 122 and performs A / D conversion on the image signal (image data) output from the image sensor 122, then transmits it to the camera MPU 125. The image processing circuit 124 performs general image processing on the image signal output from the image sensor 122, such as gamma conversion, color interpolation, and compression encoding. The image processing circuit 124 also generates signals for phase-detection autofocus, auto exposure, white balance adjustment, and subject recognition. In this embodiment, signals for phase-detection autofocus, auto exposure, white balance adjustment, and subject recognition are generated separately, but for example, the signals for auto exposure, white balance adjustment, and subject recognition may be generated as common signals. Furthermore, the combination of common signals is not limited to this.

[0020] The camera MPU (processor, control unit) 125 performs all calculations and controls related to the camera body 120. Specifically, the camera MPU 125 controls the image sensor drive circuit 123, image processing circuit 124, display 126, operation switch group 127, memory 128, phase-detection AF unit 129, AE unit 130, white balance adjustment unit 131, and subject recognition unit 132. The camera MPU 125 is connected to the lens MPU 117 via the signal lines of the mount M and communicates commands and data with the lens MPU 117. The camera MPU 125 issues requests to the lens MPU 117 for lens position acquisition and lens drive by a predetermined drive amount, and also issues requests to the lens MPU 117 for acquisition of optical information specific to the lens unit 100.

[0021] The camera MPU 125 incorporates a ROM 125a that stores a program to control the operation of the camera body 120, a RAM (camera memory) 125b that stores variables, and an EEPROM 125c that stores various parameters. The camera MPU 125 also performs focus detection processing based on the program stored in the ROM 125a. In the focus detection processing, a known correlation calculation process is performed using a pair of image signals obtained by photoelectric conversion of optical images formed by light beams passing through different pupil regions (pupil regions) of the imaging optical system.

[0022] The camera MPU 125 includes a first acquisition means 1251, a second acquisition means 1252, and a control means 1253. The first acquisition means 1251 acquires first information (estimated image plane information) regarding the first image plane position of the subject using a time-series focus detection result (defocus amount) based on the image signal obtained from the image sensor 122. The second acquisition means 1252 acquires second information (predicted image plane information) regarding the future second image plane position of the subject using the first information. The control means 1253 controls the focus lens 104 using the first information or the second information according to the movement of the subject. Note that the phase-difference AF unit 129 may have at least some of the functions (part of the functions as a control device) of each of the first acquisition means 1251, the second acquisition means 1252, and the control means 1253.

[0023] The display unit 126 consists of an LCD or the like and displays information related to the shooting mode of the imaging system 10, a preview image before shooting and a confirmation image after shooting, and an image showing the focus status when focus is detected. The operation switch group 127 consists of a power switch, a shutter button (shooting trigger), a zoom operation switch, a shooting mode selection switch, and the like. The memory (recording means) 128 is a removable flash memory that records captured images.

[0024] The phase-difference AF unit 129 performs focus detection processing using a phase-difference detection method based on the image signal (signal for phase-difference AF) of the focus detection image data obtained from the image sensor 122 and the image processing circuit 124. More specifically, the image processing circuit 124 generates a pair of image data formed by light beams passing through a pair of pupil regions of the imaging optical system as focus detection data, and the phase-difference AF unit 129 detects the amount of focus shift based on the amount of shift of the pair of image data. In this way, the phase-difference AF unit 129 of this embodiment performs phase-difference AF (image plane phase-difference AF) based on the output of the image sensor 122 without using a dedicated AF sensor.

[0025] The AE unit 130 performs exposure adjustment processing to make the shooting conditions appropriate by performing photometering based on AE signals obtained from the image sensor 122 and the image processing circuit 124. Specifically, it performs photometering based on the AE signals and calculates the exposure amount at the set aperture value, shutter speed, and ISO sensitivity. Based on the difference between the calculated exposure amount and a predetermined appropriate exposure amount, it calculates the appropriate aperture value, shutter speed, and ISO sensitivity to be set during shooting and sets them as shooting conditions to perform exposure adjustment processing. The AE unit 130 calculates the exposure conditions during shooting using the photometering results and functions as an exposure adjustment means that controls the aperture value of aperture 102, shutter speed, and ISO sensitivity.

[0026] The white balance adjustment unit 131 performs white balance adjustment processing based on white balance adjustment signals obtained from the image sensor 122 and the image processing circuit 124. Specifically, it calculates the white balance of the white balance adjustment signal and performs white balance adjustment processing by adjusting the color weights based on the difference from a predetermined appropriate white balance.

[0027] The subject recognition unit 132 performs subject recognition processing based on the subject recognition signal generated by the image processing circuit 124. Through subject recognition processing, the type and state of the subject (detection type), and the position and size of the subject (detection area) are detected. Details of the operation of the subject recognition unit 132 will be described later.

[0028] Thus, the imaging system 10 of this embodiment can perform phase-detection autofocus, metering (exposure adjustment), white balance adjustment, and subject recognition in combination. Furthermore, the imaging system 10 can select the position (image height range) for performing phase-detection autofocus, metering, and white balance adjustment according to the results of subject recognition.

[0029] (Image sensor) Next, with reference to Figures 2 and 3(a) and (b), the arrangement of imaging pixels (and focus detection pixels) of the image sensor 122 in this embodiment will be described. Figure 2 is a schematic diagram of the pixel arrangement of the image sensor 122, showing the pixel (imaging pixel) arrangement of the 2D CMOS sensor (image sensor) of this embodiment in a 4x4 range and the focus detection pixel arrangement in an 8x4 range. In this embodiment, the 2x2 pixel group 200 shown in Figure 2 has pixels 200R with spectral sensitivity of R (red) in the upper left, pixels 200G with spectral sensitivity of G (green) in the upper right and lower left, and pixels 200B with spectral sensitivity of B (blue) in the lower right. Furthermore, each pixel is composed of a first focus detection pixel 201 and a second focus detection pixel 202 arranged in a 2x1 range.

[0030] As shown in Figure 2, a large number of 4x4 pixel (8x4 focus detection pixels) are arranged on the surface to enable the acquisition of an image (focus detection signal). In this embodiment, the image sensor is described as having a pixel period P of 4 μm, a pixel count N of 5575 columns x 3725 rows = approximately 20.75 million pixels, a column-direction period PAF of focus detection pixels of 2 μm, and a focus detection pixel count NAF of 11150 columns x 3725 rows = approximately 41.5 million pixels.

[0031] Figure 3(a) is a plan view of one pixel 200G of the image sensor 122 shown in Figure 2, as seen from the light-receiving side (+z side) of the image sensor. Figure 3(b) is a cross-sectional view of the cross section aa in Figure 3(a) as seen from the -y side. In Figures 3(a) and (b), in the pixel 200G of this embodiment, a microlens 305 for focusing incident light is formed on the light-receiving side of each pixel, and a photoelectric conversion unit 301 and a photoelectric conversion unit 302 are formed, which are NH-divided (2 divisions) in the x direction and NV-divided (1 division) in the y direction. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 correspond to the first focus detection pixel 201 and the second focus detection pixel 202, respectively.

[0032] The photoelectric conversion units 301 and 302 may be PIN-structured photodiodes with an intrinsic layer sandwiched between a p-type layer and an n-type layer, or, if necessary, the intrinsic layer may be omitted and they may be pn-junction photodiodes. A color filter 306 is formed between a microlens 305 and the photoelectric conversion units 301 and 302 of each pixel. Furthermore, if necessary, the spectral transmittance of the color filter may be changed for each sub-pixel, or the color filter may be omitted.

[0033] Light incident on pixel 200G shown in Figure 3 is focused by microlens 305, spectrally separated by color filter 306, and then received by photoelectric conversion unit 301 and photoelectric conversion unit 302. In photoelectric conversion unit 301 and photoelectric conversion unit 302, electron-hole pairs are generated according to the amount of light received. After separation in the depletion layer, the negatively charged electrons are accumulated in the n-type layer (not shown), while the holes are discharged to the outside of the image sensor through the p-type layer connected to a constant voltage source (not shown). The electrons accumulated in the n-type layer (not shown) of photoelectric conversion unit 301 and photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via a transfer gate and converted into a voltage signal.

[0034] Figure 4 is a schematic diagram illustrating the correspondence between the pixel structure and pupil division of this embodiment shown in Figures 3(a) and (b). Figure 4 shows a cross-sectional view of the aa cross-section of the pixel structure of this embodiment shown in Figure 3(a) as seen from the +y side, and the pupil plane (pupil distance DS) of the image sensor. In Figure 4, the x and y axes of the cross-sectional view are inverted with respect to Figures 3(a) and (b) in order to correspond with the coordinate axes of the pupil plane of the image sensor 122.

[0035] In Figure 4, the first pupil region 501 of the first focus detection pixel 201 is substantially conjugate to the light-receiving surface of the photoelectric conversion unit 301, whose centroid is eccentric in the -x direction, by a microlens. The first pupil region 501 is the pupil region that can receive light with the first focus detection pixel 201. The centroid of the first pupil region 501 of the first focus detection pixel 201 is eccentric to the +X side on the pupil surface. In Figure 4, the second pupil region 502 of the second focus detection pixel 202 is substantially conjugate to the light-receiving surface of the photoelectric conversion unit 302, whose centroid is eccentric in the +x direction, by a microlens. The second pupil region 502 is the pupil region that can receive light with the second focus detection pixel 202. The centroid of the second pupil region 502 of the second focus detection pixel 202 is eccentric to the -X side on the pupil surface. Furthermore, in Figure 4, the pupil region 500 is the pupil region that can receive light across the entire 200G of pixels when the photoelectric conversion unit 301 and the photoelectric conversion unit 302 (first focus detection pixel 201 and second focus detection pixel 202) are combined.

[0036] In image-plane phase-detection autofocus, the pupil is divided using microlenses on the image sensor 122, and is therefore affected by diffraction. In Figure 4, the pupil distance to the pupil surface of the image sensor is several tens of millimeters, while the diameter of the microlenses is several micrometers. As a result, the aperture value of the microlenses becomes tens of thousands, causing diffraction blur on the level of several tens of millimeters. Consequently, the image on the light-receiving surface of the photoelectric conversion unit does not become a clear pupil region or pupil portion region, but rather a light-receiving sensitivity characteristic (incidence angle distribution of light-receiving rate).

[0037] Figure 5 is a schematic diagram showing the correspondence between the image sensor 122 and the pupil division in this embodiment. The light beam passing through the different pupil regions, the first pupil region 501 and the second pupil region 502, is incident on each pixel of the image sensor 122 at different angles and is received by the 2x1 divided first focus detection pixel 201 and second focus detection pixel 202. This embodiment is an example in which the pupil region is divided into two horizontally. If necessary, pupil division may be performed vertically.

[0038] In the image sensor 122 of this embodiment, a plurality of imaging pixels, each having a first focus detection pixel 201 and a second focus detection pixel 202, are arranged. The first focus detection pixel 201 receives a light beam passing through the first pupil region 501 of the imaging optical system. The second focus detection pixel 202 receives a light beam passing through a second pupil region of the imaging optical system that is different from the first pupil region 501. In addition, the imaging pixels receive a light beam passing through the pupil region, which is the sum of the first and second pupil regions of the imaging optical system.

[0039] In the image sensor 122 of this embodiment, each imaging pixel is composed of a first focus detection pixel 201 and a second focus detection pixel 202. If necessary, the imaging pixels, the first focus detection pixel 201, and the second focus detection pixel 202 may each be configured as separate pixels, and the first focus detection pixels and the second focus detection pixels may be partially arranged in a part of the imaging pixel array.

[0040] In this embodiment, a first focus signal is generated by collecting the light-receiving signals of the first focus detection pixels 201 of each pixel in the image sensor 122, and a second focus signal is generated by collecting the light-receiving signals of the second focus detection pixels 202 of each pixel to perform focus detection. Furthermore, by adding the signals of the first focus detection pixels 201 and the second focus detection pixels 202 for each pixel of the image sensor 122, an imaging signal (imaging image) with a resolution of effective pixels N is generated. However, the method of generating each signal in this embodiment is not limited to this. For example, the second focus detection signal may be generated from the difference between the imaging signal and the first focus detection signal.

[0041] (Relationship between defocus amount and image displacement amount) Next, with reference to Figure 6, the relationship between the defocus amount and image shift amount of the first focus detection signal and the second focus detection signal acquired by the image sensor 122 of this embodiment will be explained. Figure 6 is a schematic diagram showing the relationship between the defocus amount of the first focus detection signal and the second focus detection signal and the image shift amount between the first focus detection signal and the second focus detection signal.

[0042] An image sensor (not shown) of this embodiment is placed on the imaging surface 800, and, as in Figures 4 and 5, the pupil surface of the image sensor 122 is divided into a first pupil region 501 and a second pupil region 502. The amount of defocus d is defined as the distance from the imaging surface to the image-forming position of the subject, with magnitude |d| being a negative sign (d<0) for a front-focus state where the imaging position of the subject is on the subject side of the image-forming surface. A back-focus state where the imaging position of the subject is on the opposite side of the subject from the image-forming surface is defined as a positive sign (d>0). The in-focus state where the imaging position of the subject is on the image-forming surface (focus position) is d=0. In Figure 6, subject 801 shows an example of an in-focus state (d=0), and subject 802 shows an example of a front-focus state (d<0). The front-focus state (d<0) and the back-focus state (d>0) together are defined as a defocus state (|d|>0).

[0043] In the front-focused state (d<0), the light beam from the subject 802 that passes through the first pupil region 501 (second pupil region 502) is focused once, then spreads out with a width Γ1 (Γ2) centered on the centroid position G1 (G2) of the light beam, resulting in a blurred image on the image sensor 800. The blurred image is received by the first focus detection pixels 201 (second focus detection pixels 202) that constitute each pixel arranged on the image sensor, and a first focus detection signal (second focus detection signal) is generated. Therefore, the first focus detection signal (second focus detection signal) is recorded on the image sensor 800 at the centroid position G1 (G2) as a subject image with a width Γ1 (Γ2) of the subject. The blur width Γ1 (Γ2) of the subject image increases roughly proportionally with the increase in the magnitude of the defocus amount d |d|. Similarly, the magnitude of the image shift amount p (=difference in the centroid position of the light beam G1-G2) of the subject image between the first focus detection signal and the second focus detection signal, |p|, also increases roughly proportionally as the magnitude of the defocus amount d |d| increases. The same applies to the back-focused state (d>0), although the direction of the image shift of the subject image between the first focus detection signal and the second focus detection signal is opposite to that of the front-focused state.

[0044] As the amount of defocus in the imaging signal increases, either by adding the first and second focus detection signals, the magnitude of the image shift between the first and second focus detection signals increases. Therefore, the phase-difference AF unit 129 converts the amount of image shift into a detected defocus amount using a conversion coefficient calculated based on the baseline length, given the relationship that the magnitude of the image shift between the first and second focus detection signals increases with increasing defocus in the imaging signal.

[0045] (Kalman filter operation) Next, we will explain the Kalman filter operation used as an estimation means for estimating estimated image plane information (first information regarding the first image plane position of the subject), which is information corresponding to the image plane position of the subject in this embodiment.

[0046] The time series data y(k) at time k is given by the following equations (1-1) and (1-2). Time series data is also referred to as observed data. In the following explanation, time k-1, time k, and time k+1 all correspond to the times when the time series data is obtained.

[0047]

number

[0048] X(k) and m(k) are n-dimensional column vectors, A(k) is an n-dimensional column vector (state vector), and ω(k) has a mean of 0 and a variance of σ. ω 2 The observation noise is L(k), where L(k) is an n×n matrix, ν has a mean of 0, and variance σ is 0. ν 2 This is system noise. The Kalman filter operation calculates the state vector A(k), and the operation is divided into two steps: a prediction step and a filtering step.

[0049] First, the state is estimated in advance in the prediction step, and then the state is estimated using the observation results in the filtering step. In the prediction step, the prior state estimation vector A'(k) (n-dimensional column vector) and the prior error covariance matrix P'(k) (n×n matrix) are obtained by the following equations (2-1) and (2-2), respectively.

[0050]

number

[0051] As shown in equation (2), the prior state estimate vector A'(k) estimates the state vector at time k using the state vector (k-1) obtained at time k-1 and an arbitrary L(k). The prior error covariance matrix P'(k) is an equation that estimates the error between the state vector A(k) at time k and the prior state estimate vector A'(k). In the filtering step, the state estimate vector A(k) (an n-dimensional column vector) is obtained from the detected time series data y(k) using the following equation (3-1). The posterior error covariance matrix P(k) (an n×n matrix) is obtained using the following equation (3-2).

[0052]

number

[0053] As shown in equation (3-1), A(k) is the actual detection result y(k) and X is the detection result predicted in advance. T (k) is calculated by adding a correction value obtained by multiplying the difference of A'(k) by the Kalman gain g(k) to A'(k). Note that matrix I is an n×n identity matrix. The Kalman gain g(k) can be obtained by the following equation (4).

[0054]

number

[0055] As shown in equation (4), the observation noise σ ω 2 As (k) increases, g(k) decreases. Also, as the prior error covariance matrix P'(k) increases, g(k) decreases. That is, the detected y(k) and X T If it is considered highly likely that an error has occurred in (k)A'(k), then g(k) will be smaller compared to the case where no error is found. This makes the calculated A(k) less susceptible to the influence of the error. The initial value A(0) of the state vector and the initial value P(0) of the error covariance matrix are given by the following equations (5-1) and (5-2), respectively.

[0056]

number

[0057] (Kalman filter operation in this embodiment) Next, the Kalman filter operation in this embodiment will be explained. y(k) is the detection result of the image plane position at time k. In the Kalman filter operation of this embodiment, the image plane position and image plane movement velocity at time k are estimated from the state vector A(k) as information about the state of the subject. Furthermore, by obtaining the state vector A(k+1) based on the state vector A(k), the image plane position and image plane movement velocity at time k+1 are estimated as information about the state of the subject.

[0058] In this embodiment, the image plane position is the position of the rear focal point corresponding to the focus lens 104 (also referred to as the image plane position of the imaging optical system or the lens image plane position). Furthermore, the image plane position corresponding to the subject is the position of the rear focal point when the focus lens 104 is in a position where the front focal point is in focus with respect to the subject. In other words, the image plane position corresponding to the subject is the position of the rear focal point calculated by adding the amount of defocus to the position of the rear focal point at the time when focus detection is performed on the subject. In this embodiment, this is called the subject image plane position.

[0059] In this embodiment, an example is described in which the image plane position is used as the information corresponding to the image plane position, but information other than the image plane position may also be used as the information corresponding to the image plane position. For example, since the image plane position corresponds to the position of the focus lens 104, the position of the focus lens 104 corresponding to the image plane position may be used instead of the image plane position in this embodiment. In this case, the lens position corresponding to the image plane position corresponding to the subject is the focus lens position calculated as follows: That is, at the time when focus detection is performed on the subject, it is the position of the focus lens calculated by adding the amount of defocus to the focus lens position at that time.

[0060] Here, referring to FIG. 7, a model formula for predicting the movement of a subject using information regarding the state of the subject (image plane position and image plane movement speed estimated by Kalman filter operation) will be described. FIG. 7 is an explanatory diagram of Kalman filter operation in the present embodiment as an example. In FIG. 7, the horizontal axis represents time, and the vertical axis represents the image plane position, respectively.

[0061] Consider predicting the image plane position corresponding to the subject by a linear equation (two-dimensional) as shown by the solid line in FIG. 7. The image plane position at time k can be predicted by the average image plane movement speed v at time k and the image plane position y at time 0. A Let this be the model that can be predicted. At this time, define the column vector A as the image plane position (intercept) y at time 0 A and the average image plane movement speed (slope) v at time k. Also, define the column vector X as 1 such that k and y at time A become constants. The variance σ ω 2 may be set based on the variance of the detection result. In the initial value A(0), the initial value of y A may be set based on, for example, the image plane position y0 detected for the first time. Also, the initial value v of the average image plane movement speed may be set to 0. The initial value P(0) may be set to an appropriate value. The matrix L, the column vector m, and the variance σ ν 2 may be set according to the properties of the model, that is, the properties of the movement of the subject to be photographed, etc., and may be time-invariant.

[0062] Note that the image plane movement speed is the speed at which the image plane position moves and is the speed corresponding to the movement speed of the subject. In the present embodiment, the image plane movement speed is used for explanation, but it is not limited to this as long as it is the speed corresponding to the image plane movement speed. For example, it may be the movement speed of the focus lens position corresponding to the movement speed of the image plane position. In the present embodiment, an example in which the model formula is a linear equation (two-dimensional) is given as an example, but the model formula may be of any dimension according to the movement of the assumed subject, etc., and the column vector A may be defined according to the dimension of the model formula.

[0063] As described above, the matrices, vectors, and variances necessary for the prediction step have been defined. Subsequently, by repeating the filtering step using the image plane position detection results, a model equation for estimating the motion of the subject using Kalman filtering can be obtained. As mentioned above, Kalman filtering performs calculations that take errors into account, so it is possible to estimate the image plane position with good accuracy even in situations where errors are likely to occur in the focus detection results. In this embodiment, the image plane position and image plane movement velocity estimated by Kalman filtering are collectively referred to as estimated image plane information.

[0064] (Motion detection) The details of motion detection in this embodiment will be described with reference to Figures 8 and 9. Figure 8 is a flowchart of the motion detection process. Each step in Figure 8 is mainly performed by the camera MPU 125.

[0065] First, in step S801, the camera MPU 125 periodically detects focus on the subject and acquires the subject's image plane position (the position of the subject's image plane). The camera MPU 125 then determines whether the difference between the subject's image plane position at the most recent focus detection and the subject's image plane position at the previous focus detection is greater than or equal to the motion-determined image plane position difference (first threshold).

[0066] Here, the motion detection image plane position difference is a threshold used to determine if a subject has moved, based on the difference between the image plane positions of two adjacent subjects detected periodically in a time series. Figure 9 shows an example of the image plane position of a subject used for motion detection. In Figure 9, the vertical axis represents the image plane position of the subject, and the horizontal axis represents the time of focus detection. On the vertical axis, the upward direction represents the image plane position in the infinite direction, and the downward direction represents the image plane position in the nearest direction. Also in Figure 9, the black dots represent the subject's image plane position when focus is detected at each time point, and it is assumed that the focus lens is not moving while the focus detection signal for focus detection is being captured. Of the differences between adjacent subject image plane positions in a time series, differences greater than or equal to the motion detection image plane position difference are indicated by black arrows, and differences less than the motion detection image plane position difference are indicated by white arrows.

[0067] If the difference between the latest and previous subject image plane positions in step S801 is greater than or equal to the motion detection image plane position difference, the system proceeds to step S802. On the other hand, if the difference in subject image plane positions is less than the motion detection image plane position difference, the system proceeds to step S803. In step S802, the camera MPU 125 increments the motion detection counter, which is the number of times the system determined that the subject had moved in step S801. In step S803, the camera MPU 125 initializes the motion detection counter to 0 because it determined in step S801 that the subject was not moving. For example, in Figure 9, the difference in the image plane position of adjacent subjects in the most recent time series is greater than or equal to the motion detection image plane position difference for three consecutive times, so the motion detection counter becomes 3.

[0068] In step S804, the camera MPU 125 determines whether the motion detection counter is equal to or greater than the number of motion detections. If the motion detection counter is equal to or greater than the number of motion detections, the process proceeds to step S805. On the other hand, if the motion detection counter is less than the number of motion detections, the process proceeds to step S806. Here, the number of motion detections is the threshold (second threshold) for determining whether a subject is a moving subject, using the number of consecutive times that the subject was determined to have moved in step S801. For example, in Figure 9, since the motion detection counter is 3, if the number of motion detections is 3 or less, the subject is determined to be a moving subject, and the process proceeds to step S805.

[0069] In step S805, the camera MPU 125 determines that the subject is in a state where it is evaluated as moving (subject movement is the first state). In step S806, the camera MPU 125 determines that the subject is in a state where it is evaluated as not moving (subject movement is the second state). At this point, the camera MPU 125 terminates the motion detection process.

[0070] (Calculation of predicted image plane position) Referring to Figure 10, the calculation of the predicted image plane position (predicted image plane information, i.e., second information regarding the future second image plane position of the subject) in this embodiment will be explained. Figure 10 is a diagram showing an example of the image plane position of the subject used in the calculation of the predicted image plane information. In Figure 10, the vertical axis shows the image plane position of the subject, and the horizontal axis shows the time of focus detection.

[0071] In Figure 10, the black dots represent the image plane position of the subject at each time point when focus was detected, and the white dots represent the image plane position estimated by applying the subject's image plane position at each time point up to that point to a Kalman filter. Time x t This is the time when the latest focus detection signal was captured, and time x t+1 Δx is the time when the focus detection signal is next captured, and Δx is time x t and time x t+1 This is the difference, and the time lag until the next image is taken. t is time x t This is the estimated image plane position at time x. The double white dots represent the predicted image plane position in the next image, at time x t+1 The predicted image plane position in y t+1 In this embodiment, this predicted image plane position is treated as predicted image plane information. The time x estimated by the Kalman filter t The image plane movement velocity at v t Assuming this, the predicted image plane position y after a time lag Δx has elapsed is calculated using the following equation (6): t+1 It is operable.

[0072]

number

[0073] In this embodiment, the predicted image plane information (second information) was obtained using the estimated image plane information (first information) obtained from the Kalman filter, but it may also be obtained by other known methods.

[0074] (Focus adjustment processing) Next, the focus adjustment process of this embodiment will be described with reference to Figure 11. Figure 11 is a flowchart of the focus adjustment process. Each step in Figure 11 is mainly performed by the parts of the imaging system 10 in accordance with the commands of the camera MPU 125.

[0075] First, in step S1101, the camera MPU 125 performs focus detection processing. Specifically, the camera MPU 125 uses the image sensor 122 and the phase-difference AF unit 129 to capture an image signal and a focus detection signal, and uses these signals to perform focus detection and calculate the amount of defocus and the position of the subject image plane.

[0076] Next, in step S1102, the camera MPU 125 performs subject recognition processing. That is, the camera MPU 125 uses the subject recognition unit 132 to recognize a specific subject, such as a person or an animal, from the image signal obtained in step S1101. Next, in step S1103, the camera MPU 125 performs recording processing. The time-series subject image plane position obtained in step S1101 is recorded in the memory 128. Next, in step S1104, the camera MPU 125 performs motion detection processing. That is, the camera MPU 125 performs motion detection based on the time-series subject image plane position recorded in step S1103, as explained with reference to Figure 8.

[0077] Next, in step S1105, the camera MPU 125 performs a Kalman filter operation. That is, the camera MPU 125 performs the aforementioned Kalman filter operation based on the time-series subject image plane position recorded in step S1103, and estimates the image plane position and image plane movement velocity, which are the estimated image plane information (first information). Next, in step S1106, the camera MPU 125 calculates the predicted image plane information (second information). That is, the camera MPU 125 calculates the aforementioned predicted image plane information based on the estimated image plane information and the time lag until the next imaging time (the period from the last imaging time to the next imaging time).

[0078] Next, in step S1107, the camera MPU 125 determines whether the subject is moving or not based on the result of the motion detection process in step S1104. If the subject is moving (the subject's movement is in the first state), the process proceeds to step S1108. On the other hand, if the subject is not moving (the subject's movement is in the second state), the process proceeds to step S1109.

[0079] In step S1108, the camera MPU 125 controls the focus lens 104 based on the predicted image plane information. That is, the camera MPU 125 drives the focus lens 104 toward the predicted image plane position, which is the predicted image plane information acquired in step S1106.

[0080] In step S1109, the camera MPU 125 determines whether the focus state is in focus or not. For example, if the amount of defocus acquired in step S1101 is less than or equal to a predetermined amount, the camera MPU 125 determines that the focus state is in focus. On the other hand, if the amount of defocus is greater than the predetermined amount, it determines that the focus state is not in focus. If the focus state is in focus, the focus adjustment process ends. If the focus state is not in focus, the process proceeds to step S1110.

[0081] In step S1110, the camera MPU 125 determines whether or not estimated image plane information can be used. For example, if the subject image plane position in the time series applied to the Kalman filter is greater than or equal to a predetermined number of history points and estimated image plane information can be used, the process proceeds to step S1111. On the other hand, if the subject image plane position in the time series is less than a predetermined number of history points and estimated image plane information cannot be used, the process proceeds to step S1112.

[0082] In step S1111, the camera MPU 125 controls the focus lens 104 using the estimated image plane information. That is, the camera MPU 125 drives the focus lens 104 toward the image plane position estimated in the Kalman filter calculation in step S1103.

[0083] In step S1112, the camera MPU 125 controls the focus lens 104 using the subject image plane position. That is, the camera MPU 125 drives the focus lens 104 toward the subject image plane position acquired by the focus detection process in step S1101. In other words, if the control means 1253 cannot acquire first information using a predetermined number of time-series focus detection results, it controls the focus lens 104 using third information regarding the third image plane position of the subject acquired using the last acquired focus detection result. After executing steps S1108, S1111, and S1112, the camera MPU 125 processes the next image again starting from step S1101.

[0084] Based on the above, subjects with large changes in focus position are determined to be moving subjects. In this case, the focus tracking performance can be improved by controlling the focus lens 104 based on the predicted image plane information. On the other hand, subjects with small changes in focus position are determined to be stationary subjects. In this case, the focus lens 104 can be controlled based on the estimated image plane information. By using the estimated image plane information, the Kalman filter can track focus while considering the change in the image plane position of the subject, and stabilize the focus position even in situations where errors are likely to occur in the focus detection result.

[0085] Furthermore, in the focus determination in step S1109, focus may be determined to be achieved if the difference between the estimated image plane position and the lens image plane position is less than or equal to a predetermined difference. This allows the focus position to be stabilized by stopping the drive of the focus lens 104, even in situations where errors are likely to occur in the focus detection result. In addition, the Kalman filter has the property that the estimated image plane information does not converge and fluctuates when the subject image plane position in the applied time series is small. For this reason, if the subject image plane position in the time series applied to the Kalman filter is less than a predetermined number of history entries, focus may be determined to be lost. This prevents incorrect determination of focus due to estimated image plane information before convergence or a defocus amount below a predetermined amount that is incorrectly detected in situations where errors are likely to occur in the focus detection result.

[0086] In the Kalman filter calculation in step S1105, the parameters used in the Kalman filter calculation may be set (changed) according to the motion determination process in step S1104. That is, the control means 1253 may change the parameters used to acquire estimated image plane information according to the movement of the subject.

[0087] As mentioned above, in the Kalman filter operation, the matrix L and column vector m should be set according to the properties of the subject being photographed, such as its movement. Therefore, by determining that the properties of the subject's movement have changed based on the motion detection result and setting the matrix L and column vector m accordingly, more accurate estimated image plane information can be obtained that matches the subject's movement. Alternatively, the type of subject recognized in the subject recognition process of step S1102 may be used to determine that the properties of the subject's movement have changed, and the matrix L and column vector m may be set accordingly. In other words, the control means 1253 may change the parameters used to acquire estimated image plane information according to the type of subject.

[0088] If the subject recognized in the subject recognition process in step S1102 changes, the recording process in step S1103 may discard the previously recorded time-series subject image plane position and record the subject image plane position from the image taken after the subject changed. That is, if the subject switches from the first subject to the second subject, the first acquisition means 1251 may acquire estimated image plane information using the time-series focus detection results for the second subject acquired after the switch.

[0089] Alternatively, the state estimation vector A(k) and posterior error covariance matrix P(k) calculated in the Kalman filter operation in step S1105 may be discarded at the same time, and the Kalman filter operation may be performed again from the initial value A(0) of the state vector and the initial value P(0) of the error covariance matrix. This prevents the situation where, when the subject is switched, the image plane position of the subject before the switch affects the calculation of an appropriate estimated image plane position for the subject after the switch.

[0090] The imaging system 10 of this embodiment may be capable of both video recording and still image recording. In this embodiment, the control means 1253 does not need to use estimated image plane information when determining focus and controlling the focus lens 104 when the shooting state of the imaging system 10 is still image shooting. That is, when the shooting state is still image shooting, the determination using estimated image plane information in step S1109 is prohibited, and the determination of whether or not to use estimated image plane information in step S1110 is always set to "NO", thereby prohibiting focus lens control based on estimated image plane information. This prevents a decrease in AF responsiveness during still image shooting, where AF responsiveness is required more than during video recording, when the estimated image plane information lags behind the actual movement of the subject due to the influence of the subject's image plane position over time.

[0091] In the recording process of step S1103, estimated image plane information calculated using a Kalman filter based on the previous image may be recorded in time series, and in the motion determination of step S1104, the time-series estimated image plane position may be used instead of the time-series subject image plane position for determination. This reduces the likelihood of erroneous motion determination results by using highly accurate estimated image plane information, even in situations where errors are likely to occur in the focus detection result.

[0092] As described above, in this embodiment, the control means 1253 changes the information used when controlling the focus lens 104 depending on whether the subject is moving or not. For example, if the subject is in a first state (a state in which it is evaluated as not moving), the control means 1253 controls the focus lens 104 using estimated image plane information (first information). On the other hand, if the subject is in a second state (a state in which it is evaluated as moving) which is greater than the first state, the control means 1253 controls the focus lens 104 using predicted image plane information (second information). Preferably, the subject's movement is in the direction of the optical axis of the imaging optical system.

[0093] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0094] According to each embodiment, it is possible to improve focusing stability even when the amount of defocus varies, while appropriately tracking a moving subject. Therefore, according to each embodiment, it is possible to provide a control device, imaging device, control method, and program that enable stable focusing.

[0095] Each embodiment of the disclosure includes the following configuration and method. (Composition 1) A first acquisition means that acquires first information regarding the first image plane position of a subject using a time-series focus detection result based on the image signal obtained from the image sensor, A second acquisition means for acquiring second information relating to the future second image plane position of the subject using the first information, A control device comprising: control means for controlling a focus lens using the first information or the second information in accordance with the movement of the subject. (Configuration 2) The control means is When the movement of the subject is in a first state, the focus lens is controlled using the first information. The control device according to configuration 1, characterized in that when the movement of the subject is greater than the first state, it controls the focus lens using the second information. (Composition 3) The control device according to configuration 2, characterized in that the control means performs a focus determination using the first information when the movement of the subject is in the first state. (Composition 4) The control device according to configuration 3, characterized in that the control means does not use the first information when determining focus and controlling the focus lens when the shooting state is a still image shooting state. (Composition 5) The control device according to any one of configurations 1 to 4, characterized in that the first information includes information regarding the image plane movement velocity estimated using the time-series focus detection results. (Composition 6) The control device according to any one of configurations 1 to 4, characterized in that the first information includes information regarding the image plane position estimated using the time-series focus detection results.

[0096] (Composition 7) The control device according to any one of configurations 1 to 6, characterized in that the second information is predicted based on the first information and the period from the last imaging time to the next imaging time. (Composition 8) The control device according to any one of configurations 1 to 7, characterized in that the movement of the subject is movement in the optical axis direction of the imaging optical system. (Composition 9) The control device according to any one of configurations 1 to 8, characterized in that the control means changes the parameters used to acquire the first information in accordance with the movement of the subject. (Composition 10) The control device according to any one of configurations 1 to 9, characterized in that the control means changes the parameters used to acquire the first information according to the type of subject. (Composition 11) The control device according to any one of configurations 1 to 10, characterized in that when the subject recognized using the image signal switches from the first subject to the second subject, the first acquisition means acquires the first information using the time-series focus detection results for the second subject acquired after the switch. (Composition 12) The control device according to any one of configurations 1 to 11, characterized in that, if the control means cannot acquire the first information using the time series of focus detection results for a predetermined number of history points or more, it controls the focus lens using third information relating to the third image plane position of the subject acquired using the last acquired focus detection result. (Composition 13) The control device according to any one of configurations 1 to 12, characterized in that the control means determines the movement of the subject using the first information. (Composition 14) An imaging device characterized by having a control device according to any one of configurations 1 to 13 and the image sensor. (Method 1) A step of acquiring first information regarding the position of the subject on the first image plane using a time-series focus detection result based on the image signal obtained from the image sensor, A step of obtaining second information regarding the future second image plane position of the subject using the first information, A control method characterized by comprising the step of controlling a focus lens using the first information or the second information in accordance with the movement of the subject. (Composition 15) A program characterized by causing a computer to execute the control method described in Method 1.

[0097] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of symbols]

[0098] 122 Image sensor 125 Camera MPU (Control Unit) 1251 First acquisition method 1252 Second acquisition method 1253 Control means

Claims

1. A first acquisition means that acquires first information regarding the first image plane position of a subject using a time-series focus detection result based on an image signal obtained from an image sensor, A second acquisition means for acquiring second information relating to the future second image plane position of the subject using the first information, A control device characterized by having control means for controlling a focus lens using the first information or the second information in accordance with the movement of the subject.

2. The control means is When the movement of the subject is in a first state, the focus lens is controlled using the first information. The control device according to claim 1, characterized in that when the movement of the subject is greater than the first state, the control device controls the focus lens using the second information.

3. The control device according to claim 2, characterized in that the control means performs a focus determination using the first information when the movement of the subject is in the first state.

4. The control device according to claim 3, characterized in that the control means does not use the first information when determining focus and controlling the focus lens when the shooting state is a still image shooting state.

5. The control device according to claim 1, characterized in that the first information includes information regarding the image plane movement velocity estimated using the time-series focus detection results.

6. The control device according to claim 1, characterized in that the first information includes information regarding the image plane position estimated using the time-series focus detection results.

7. The control device according to claim 1, characterized in that the second information is predicted based on the first information and the period from the last imaging time to the next imaging time.

8. The control device according to claim 1, characterized in that the movement of the subject is movement in the optical axis direction of the imaging optical system.

9. The control device according to claim 1, characterized in that the control means changes the parameters used to acquire the first information in accordance with the movement of the subject.

10. The control device according to claim 1, characterized in that the control means changes the parameters used to acquire the first information according to the type of subject.

11. The control device according to claim 1, characterized in that when the subject recognized using the image signal switches from the first subject to the second subject, the first acquisition means acquires the first information using the time-series focus detection results for the second subject acquired after the switch.

12. The control device according to claim 1, characterized in that, if the control means cannot acquire the first information using the time series of focus detection results that exceed a predetermined number of histories, it controls the focus lens using the third information relating to the third image plane position of the subject acquired using the last acquired focus detection result.

13. The control device according to claim 1, characterized in that the control means determines the movement of the subject using the first information.

14. An imaging device comprising a control device according to any one of claims 1 to 13 and the image sensor.

15. A step of acquiring first information regarding the position of the subject on the first image plane using a time-series focus detection result based on the image signal obtained from the image sensor, A step of obtaining second information regarding the future second image plane position of the subject using the first information, A control method characterized by comprising the step of controlling a focus lens using the first information or the second information in accordance with the movement of the subject.

16. A program characterized by causing a computer to execute the control method described in claim 15.

Images