Imaging device

Abstract

To provide an imaging apparatus that can achieve a precise autofocus function.SOLUTION: An imaging apparatus 1 comprises: an image sensor 11 in which two types of phase difference detection pixels different in opening state are arranged; a phase difference information reading unit 12 that reads out, from the image sensor 11, pixel values of the phase difference detection pixels having the same opening state and creates two phase difference images; a phase difference detection unit 14 that detects a detection phase difference being the phase difference between the two phase difference images; a reference phase difference conversion unit 15 that, with a predetermined operational expression using the ratio between a lens parameter during imaging and a reference parameter being a lens parameter to be a standard, converts the detection phase difference into a reference phase difference when imaging is performed with the reference parameter; and a focus control unit 16 that controls focus with a preset focus drive amount corresponding to the reference phase difference.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device using a phase difference detection technique. [Background technology]

[0002] In recent years, many imaging devices have been equipped with an autofocus function that automatically performs lens focusing operations. To achieve autofocus, a technology that detects the difference (defocus amount) between the position of a subject to be focused and the focus position of the lens is important. A well-known defocus amount detection technology is phase difference detection technology. Phase difference detection technology is a method of calculating the defocus amount by pupil-dividing the light incident on an imaging element, acquiring images (phase images) corresponding to each divided region, and detecting the spatial relative distance (phase difference) between these phase images (see, for example, Patent Document 1).

[0003] The technique described in Patent Document 1 is an example of an autofocus technique that uses image plane phase difference detection technology with a phase image acquisition function provided on an image sensor. In this example, the exit pupil is divided into regions using multiple types of image plane phase difference pixels, and the image plane phase difference pixels corresponding to the same divided regions are used to generate a group of phase images, and the defocus amount is calculated from the phase difference of the group of phase images, and the focus drive amount of the lens is determined from the defocus amount, thereby realizing the autofocus function. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-99416 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 does not explicitly state the procedure for calculating the defocus amount from the phase difference calculated from the image of the image plane phase difference pixel, or the procedure for determining the focus drive amount of the lens. Generally, the phase difference obtained by phase difference detection technology for the defocus amount depends on parameters (lens parameters) such as the subject position, lens focus position, focal length, and F-number. Therefore, to accurately calculate the defocus amount from the phase difference, it is necessary to convert the phase difference to the defocus amount using this information. Under natural image capturing conditions, the subject position and lens focus position range from several meters to several hundred meters, and the focal length ranges from several tens of millimeters to several hundred millimeters. In other words, the conversion from phase difference to defocus amount must have sufficient digit accuracy to accommodate a wide range.

[0006] However, when an image capture device with an autofocus function attempts to calculate the defocus amount using an arithmetic circuit with limited computational resources, it becomes difficult to obtain a sufficient number of digits, resulting in errors in the calculation results. That is, problems occur in the autofocus function, such as a hunting phenomenon caused by overestimating the focus drive amount, or a decrease in focusing speed caused by underestimating the focus drive amount.

[0007] The present invention has been made in consideration of the above-mentioned conventional problems, and an object of the present invention is to provide an imaging device that can achieve an accurate autofocus function while reducing calculation resources. [Means for solving the problem]

[0008] In order to solve the above problem, the imaging device of the present invention is an imaging device that captures images through a focus-controllable lens, and is configured to include an image sensor, a phase difference information readout unit, a phase difference detection unit, a reference phase difference conversion unit, and a focus control unit.

[0009] In this configuration, the imaging device reads out pixel values ​​of phase difference detection pixels having the same type of opening state from an image sensor in which two types of phase difference detection pixels having different opening states are arranged at specific pixels using a phase difference information readout unit, and generates two phase difference images. These two phase difference images are images of the same subject captured with different openings, and therefore have a phase difference. Then, the imaging device detects the phase difference between the two phase difference images by block matching or the like using a phase difference detection unit.

[0010] Furthermore, the imaging device uses the ratio between the lens parameters at the time of imaging and the reference parameters, which are reference lens parameters, to convert the phase difference detected by the phase difference detection unit into the phase difference (reference phase difference) when imaging is performed with the reference parameters, according to a predetermined calculation formula, by using the ratio between the lens parameters and the reference parameters. In this way, the reference phase difference conversion unit can convert the phase difference into the phase difference when imaging is performed with the reference parameters, while reducing calculation resources, by using the same type of ratio between the lens parameters and the reference parameters. The imaging device then controls the focus by the focus control unit at a preset focus drive amount corresponding to the phase difference when imaging is performed using the reference parameters. In this way, the focus control unit can control the focus in response to only the phase difference based on the reference parameters. [Effects of the Invention]

[0011] The present invention provides the following excellent effects. According to the present invention, without directly calculating the defocus amount, the phase difference is converted from the same ratio between the lens parameters at the time of image capture and the reference parameters serving as a standard into the phase difference when image capture is performed using the reference parameters, and focus can be controlled using a focus drive amount that is preset based on the reference parameters. As a result, the present invention can achieve an autofocus function with high accuracy while reducing the computational resources. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 2] 2A and 2B are diagrams showing the configuration of an image sensor of the imaging device of FIG. 1, in which (a) is a front view and (b) is a side view. [Figure 3] 2A and 2B are structural diagrams of pixel units corresponding to phase difference detection pixels in the image sensor of the imaging device of FIG. 1, where FIG. 2A shows a pixel structure with an opening in the left half, and FIG. 2B shows a pixel structure with an opening in the right half. [Figure 4] 10A and 10B are diagrams illustrating an example of a color structure of an imaging surface captured by an image sensor in which phase difference detection pixels are formed in a part of G pixels. [Figure 5] 10A and 10B are explanatory diagrams for explaining a geometric model for calculating a phase difference and a defocus amount. [Figure 6] FIG. 10 is a graph showing the relationship between the phase difference and the defocus amount. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of a lookup table. [Figure 8] 4 is a flowchart illustrating an operation of the imaging device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] [Configuration of imaging device] First, with reference to FIG. 1, the configuration of an imaging device 1 according to an embodiment of the present invention will be described. The imaging device 1 captures an image through a focus-controllable lens 10. In particular, the imaging device 1 uses an autofocus function to capture an image of a subject in focus with high precision. As shown in FIG. 1, the imaging device 1 includes a lens 10, an image sensor 11, a phase difference information reading unit 12, a sensitivity correction unit 13, a phase difference detection unit 14, a reference phase difference conversion unit 15, a focus control unit 16, a pixel interpolation unit 17, and an image processing unit 18.

[0014] The lens 10 forms an image of a subject on the imaging surface of the image sensor 11. A typical camera lens made up of a single or multiple optical lenses can be used as the lens 10. The lens 10 may be configured to be separable from the imaging device 1. Here, the lens 10 is provided with a lens driver 10a and is focus-controlled. The lens driver 10a may be built into the body of the lens 10 or may be an external module that can be separated from the body of the lens 10.

[0015] The lens driver 10a drives the focus position of the lens 10. The lens driver 10a moves the focus position of the lens 10 in the optical axis direction according to a driving amount output from a focus control unit 16, which will be described later. The lens driver 10a may also have a function of driving the focal length, iris (F-number), etc. of the lens 10. The lens driver 10a also has a function of outputting lens parameters to the outside, such as the focus position, focal length, and F-number of the lens 10.

[0016] The image sensor 11 has at least two types of phase difference detection pixels arranged at a specific pixel for performing image plane phase difference detection with different aperture states, where the aperture state refers to the position, shape, or both of the aperture states. The image sensor 11 converts light incident via the lens 10 into an image. Note that the following description will be given taking an example of a single-chip imaging type image sensor.

[0017] The structure of the image sensor 11 will now be described with reference to FIGS. As shown in FIG. 2, the image sensor 11 includes a microlens array 11a, a color filter 11b, a wiring layer 11c having an opening, and a photodiode 11d.

[0018] The microlens array 11a is a group of lenses in which microlenses (convex lenses) are arranged two-dimensionally. Each microlens in the microlens array 11a corresponds to a pixel, and incident light is irradiated onto a photodiode 11d arranged at the focal length of the microlens.

[0019] The color filter 11b has a narrow transmission spectrum that transmits light of a specific color wavelength. Here, the color filter 11b is an RG1G2B Bayer array filter with a ratio of red (R), green (G), and blue (B) of 1:2:1, but other colors and arrays may also be used.

[0020] The wiring layer 11c is a layer on which wiring is formed for inputting and outputting voltage to and from the photodiode 11d. In this wiring layer 11c, a transparent region where no wiring is formed is an effective pixel region, which guides incident light to the photodiode 11d.

[0021] The photodiode 11d converts incident light into an electrical signal, and the incident surface of the photodiode 11d serves as an imaging surface for capturing an image. Here, the opening state of the wiring layer 11c is changed for the phase difference detection pixel corresponding to a predetermined position, and light to the photodiode 11d is partially blocked.

[0022] For example, as shown in Fig. 3, the wiring layer 11c blocks 50% of light on each side. S Opening H L 3(b) shows an example in which the wiring layer 11c is formed so that light is irradiated by the right side R of the photodiode 11d. S Opening H R In this example, the wiring layer 11c is formed so that light is irradiated by the wiring layer 11c. In order to form a light-shielding region, a dedicated light-shielding plate may be provided directly above the photodiode 11d. Furthermore, the phase difference detection pixels are of the same color on the imaging surface, for example, some or all of the G pixels. Of course, the R pixels and the B pixels may be used as phase difference detection pixels. Furthermore, the color filters corresponding to the phase difference detection pixels may be transparent filters. Furthermore, although the opening positions are changed in the horizontal direction here, the opening positions may also be changed in the vertical direction.

[0023] FIG. 4 shows an example of the color structure of the imaging surface captured by the image sensor 11, in which phase difference detection pixels are formed in part of the G pixels on the imaging surface. FIG. 4 shows the phase difference detection pixels (left eye phase difference detection pixel L) that are configured by arranging some of the G (G2) pixels as phase difference detection pixels and blocking incident light from the left and right eyes. G and right-eye phase difference detection pixel R G ) are arranged alternately at predetermined intervals in two dimensions. In this way, the image sensor 11 generates a captured image made up of pixel values ​​of the pixels of each color and the phase difference detection pixels from the incident light. Returning to FIG. 1, the description of the configuration of the imaging device 1 will continue. The image sensor 11 outputs the captured image to the phase difference information reading unit 12 .

[0024] The phase difference information reading unit 12 reads out the pixel values ​​of the phase difference detection pixels having the same type of opening state from the image sensor 11, and generates two phase difference images that serve as phase difference information. Here, the phase difference information reading unit 12 includes a region of interest cutting unit 12a.

[0025] The region-of-interest cutting unit 12a cuts out a region of interest from each of the two phase difference images read by the phase difference information reading unit 12. The region of interest is not particularly limited, but may be, for example, a region manually specified from outside. In this case, the operator recognizes the output image of the imaging device 1 via a monitor (not shown) and specifies a subject or the like on which the operator particularly wants to focus using a rectangular region or the like. Of course, this region of interest may be identified using an existing machine-learned model for person or object detection, etc. Alternatively, the region of interest may be a predetermined region, such as the center of the captured image.

[0026] The attention area cutting unit 12a cuts out the images of the phase difference detection pixels from the attention area of ​​the captured image, and generates phase difference images (left eye phase difference image L, right eye phase difference image R). For example, when the image sensor 11 has the pixel structure shown in FIG. 4, the attention area cutout unit 12a extracts the left-eye phase difference detection pixel L G The attention area cutout unit 12a cuts out an image consisting of only the right-eye phase difference detection pixel R as the left-eye phase difference image L. G An image consisting of only the right eye phase difference image R is extracted from the captured image. The attention area cutout unit 12a outputs the generated left eye phase difference image L and right eye phase difference image R to the sensitivity corrector 13. Furthermore, the phase difference information reading unit 12 outputs to the pixel interpolation unit 17 an image captured by the image sensor 11 in which the phase difference detection pixel is a missing pixel.

[0027] The sensitivity correction unit 13 corrects the sensitivity difference (brightness difference, contrast difference) between the phase difference images (left eye phase difference image L, right eye phase difference image R) generated by the phase difference information reading unit 12 (region of interest cutting unit 12a).

[0028] For example, the sensitivity correction unit 13 normalizes each image so that the peak values ​​(maximum and minimum values) of the sensitivities of the left-eye phase-difference image L and the right-eye phase-difference image R match. Alternatively, the sensitivity correction unit 13 may correct the sensitivity difference between the left-eye phase-difference image L and the right-eye phase-difference image R using a general histogram matching method. Alternatively, the sensitivity correction unit 13 may correct the sensitivity difference between the left-eye phase-difference image L and the right-eye phase-difference image R by multiplying the images by a shading correction coefficient determined from spatial pixel sensitivity characteristics that are designed or measured in advance of the phase difference detection pixels. The sensitivity corrector 13 outputs the corrected left-eye phase-difference image L′ and right-eye phase-difference image R′ to the phase-difference detector 14 .

[0029] The phase difference detection unit 14 detects the phase difference P between the sensitivity-corrected left-eye phase difference image L′ and the sensitivity-corrected right-eye phase difference image R′ (hereinafter referred to as the detected phase difference) dif The detected phase difference P dif can be calculated as the magnitude of the relative position vector.

[0030] For example, the phase difference detection unit 14 performs block matching between the left-eye phase difference image L′ and the right-eye phase difference image R′ to determine the magnitude of the relative position vector. Note that the search direction of the blocks in the block matching is set to conform to the arrangement direction (light blocking direction) of the aperture. For example, in this case, the phase difference image is obtained by performing block matching between the left-eye phase difference detection pixel L′ and the right-eye phase difference image R′ as shown in FIG. G and right-eye phase difference detection pixel R G Therefore, the phase difference detection unit 14 sets the search direction for block matching to the horizontal direction.

[0031] Furthermore, when performing block matching, the phase difference detection unit 14 uses subpixel units, which are smaller than pixels, as search units, which allows the phase difference detection unit 14 to detect phase differences more finely and accurately than the pixel units of the image sensor 11. The phase difference detector 14 detects the phase difference P dif is output to the reference phase difference conversion unit 15.

[0032] The reference phase difference conversion unit 15 converts the detected phase difference into a phase difference (hereinafter referred to as a reference phase difference) when an image is captured using lens parameters (reference parameters) that serve as a predetermined reference. The reference phase difference conversion unit 15 reads out the lens parameters from the lens driver 10a, and converts the conversion coefficient according to the lens parameters into the detected phase difference P detected by the phase difference detection unit 14. dif By multiplying this, the reference phase difference P ref Calculate. This conversion coefficient is a coefficient calculated from the ratio between each value of the lens parameter at the time of image capture included in an approximation function that approximates the relationship between the phase difference in image plane phase difference detection and the defocus amount using a curve based on a geometric model of the lens, and each value of a predetermined reference lens parameter (reference parameter). Specifically, the reference phase difference conversion unit 15 converts the reference phase difference P ref is calculated using the following formula (1).

[0033]

number

[0034] In the above formulas (1) and (2), Corr represents a conversion coefficient. Furthermore, Fn is the F-number (ratio of focal length to effective aperture) of the lens 10, f is the focal length of the lens 10, and L is the focus distance of the lens 10, which are actual lens parameters at the time of image capture. Also, Fn ref , f ref , L ref indicates the reference parameters F-number, focal length, and focus distance, respectively.

[0035] Here, the reference parameters are lens parameters that serve as predetermined references, and are preset as arbitrary values ​​that can be taken by lenses that are actually used. Specifically, these are lens parameters that are expected to be frequently used as lens parameters of lenses that are actually used, median values ​​of lens parameters of lenses that are actually used, etc. In addition, in equation (2), the exponents a, b, and c are any real numbers, but it is preferable to set a=1, b=−2, and c=2 based on the correspondence of the approximation function that indicates the relationship between the phase difference (detection phase difference) and the defocus amount. That is, it is preferable that the conversion coefficient Corr is calculated by the following equation (3).

[0036]

number

[0037] In this way, the bases of the exponents in equations (2) and (3) are ratios of values ​​with similar numbers of digits, and therefore rarely take extremely large or small values. Therefore, the reference phase difference conversion unit 15 can obtain the conversion coefficients using an arithmetic circuit with a small number of digits.

[0038] Here, the conversion coefficient will be described with reference to FIGS. 5 is a diagram showing a geometric model that schematically represents the deviation (defocus amount) between the focus distance of the lens and the distance to the actual subject (subject distance) when using phase difference pixels whose lower half in the vertical direction of the page is light-shielded, and the phase difference relative to the defocus amount. Here, the lens 10 is a convex lens that schematically represents the optical system. The geometric model in Figure 5 shows the detected phase difference P dif can be calculated by the following formula (4), and the defocus amount X can be calculated by the following formula (5).

[0039]

number

[0040] The geometric model in Fig. 5 is a model that shows the optical path of light rays from the subject O that pass through the aperture decenter of the exit pupil (effective aperture determined by the diaphragm) of the lens 10 until the image height of the imaging plane of the image sensor 11 reaches "0 (zero)," i.e., until the image is formed in the center. Note that the phase difference pixels of the image sensor 11 are equivalent to half of the lens 10 being 50% light-shielded. Furthermore, the ideal aperture decenter that is uniquely determined by the aperture shape of the lens 10 (the aperture shape of the phase difference pixels) is denoted as A.

[0041] D in the above formula (4) is the imaging position (distance from the center of the sensor) of the image sensor 11 of the light beam that passes through the aperture decenter of the lens 10 from the object O' in an out-of-focus state. Here, when a combination of the phase difference pixel of FIG. 5 and a phase difference pixel in which the opposite side (upper half of the top and bottom of the paper) of FIG. 5 is shielded is considered as a pair of phase difference pixels, the detected phase difference P dif can be calculated by the above-mentioned formula (4). Note that the pixel pitch p of the phase difference pixels is known information that is set in advance.

[0042] In addition, in the formulas (5) to (8), L D is the focus distance of the lens 10 from the object O to the imaging surface of the image sensor 11 when the object is in focus, and T D indicates the subject distance from the subject O' to the imaging surface of the image sensor 11 in an out-of-focus state. Also, L obj is the distance from the object O to the principal point of the lens 10, T obj indicates the distance from the object O' to the principal point of the lens 10. Also, L img is the distance from the principal point of the lens 10 to the imaging surface of the image sensor 11, T img indicates the distance from the principal point of the lens 10 to the optical axis of the lens 10 where the subject light reaches. Furthermore, f represents the focal length of the lens 10.

[0043] Furthermore, the aperture decentering A in the formula (6) is the center of gravity of the aperture area of ​​the lens 10, and since 50% of the light is blocked in this example, A=0.25×f / Fn, where Fn is the ratio between the focal length of the lens 10 and the exit pupil (effective aperture). In the above formula (1), T D In contrast to L img Assuming that is sufficiently small, the approximate result is shown in equation (9).

[0044]

number

[0045] Here, if we further consider that the second term of Equation (9) is sufficiently small compared to the first term, the reference phase difference P ref is given by the first term of equation (9), 2 is proportional to Fn,L D 2 It can be seen that it is inversely proportional to Therefore, the reference phase difference conversion unit 15 converts the reference parameters (Fn) to the lens parameters (Fn, f, L) at the time of image capture as shown in equations (2) and (3). ref ,f ref ,L ref ) is multiplied by the inverse of the ratio, and the conversion coefficient is used to convert the detected phase difference that varies depending on the lens parameters into a reference phase difference when the reference parameters are used.

[0046] 6 is a graph showing the relationship between the defocus amount (horizontal axis) and the phase difference (vertical axis). D is set to 16 m. Also, the lens 10 is assumed to be a zoom lens whose focal length can be changed from 15 mm to 480 mm. MOD (Minimum Object Distance) indicates the shortest shooting distance of the lens 10, and is set to 3 m here. Graphs A1 and A2 in FIG. 6 before conversion show the relationship between the defocus amount calculated from the geometric model of lens 10 (Equations (4) to (8), FIG. 5)) and the phase difference (detected phase difference) when the focal length f is changed. Graph B in Figure 6 shows the relationship between the defocus amount calculated from equation (1) using the conversion coefficient of equation (3) and the phase difference (reference phase difference) when the focal length f is changed. In equation (3), Fn is 2.8, f ref is 120mm, Fn ref is 4.0, L ref is set at 12m.

[0047] As is clear from FIG. 6, the relationship between the defocus amount and the phase difference before conversion is highly dependent on the focal length f, and the range of values ​​is wide. On the other hand, the relationship between the defocus amount and the converted phase difference varies little, regardless of the focal length f. In this way, by multiplying the phase difference (detected phase difference) by the conversion coefficient, it can be seen that the curve showing the relationship between the detected phase difference obtained from the lens parameters at the time of image capture and the defocus amount is projected with sufficient approximation to the curve showing the relationship between the reference phase difference obtained from the reference parameters obtained by image capture of the same subject and the defocus amount. In this way, the reference phase difference conversion unit 15 can obtain a phase difference as if an image was captured using standard reference parameters, regardless of the lens parameters used during image capture. Returning to FIG. 1, the description of the configuration of the imaging device 1 will continue. The reference phase difference conversion unit 15 converts the converted reference phase difference P ref to the focus control unit 16.

[0048] The focus control unit 16 converts the reference phase difference P ref The focus of the lens 10 is controlled based on the above. The focus control unit 16 calculates the reference phase difference P ref The focus driving signal (focus driving amount) for the lens 10 is generated according to the above and output to the lens driver 10a. The focus control unit 16 calculates the reference phase difference P ref The direction of focus drive (image sensor side or infinity side) is determined by the sign of ref The focus drive amount (movement amount or drive speed) is determined by the absolute value of

[0049] The correspondence relationship between the absolute value of the reference phase difference and the focus drive amount may be stored in advance as a lookup table (LUT) in a memory (not shown) and referenced by the focus control unit 16. It is preferable to set in advance in this lookup table a focus drive amount that provides optimal focus control for the defocus amount determined from the phase difference (reference phase difference) when imaging is performed under the conditions of the reference parameters.

[0050] As explained in Figure 6, graph B after correction using the conversion coefficient has an extreme value at a specific position when the defocus amount sign is in the negative range and the focus position is in front of the subject (front-focus range). Therefore, the same phase difference exists for different defocus amounts on the front-focus side. In this case, if the same phase difference is associated with the same focus drive amount, the sign of the displacement will be reversed around the extreme value, causing the focus drive to temporarily slow down and then accelerate again, which may result in unstable movement of the focus drive amount. Therefore, it is preferable to set the focus drive amount in stages for each predetermined section in the front focus area in the lookup table.

[0051] FIG. 7 shows an example of the lookup table settings. As shown in FIG. 7, the lookup table is set so that the focus drive amount monotonically increases as the phase difference increases in a region where the focus position is deeper than the subject and the sign of the phase difference is positive (back focus region). Furthermore, in the lookup table, the focus drive amount is set in stages in the area where the phase difference sign is negative (front focus area). Here, as shown in Fig. 7, the focus drive amount is set in stages so that it remains constant at least in the area from near the extreme value of graph B in Fig. 6 to the MOD section (approximately -6 m to -13 m). This allows the focus control unit 16 to stably drive the focus. Note that the lookup table settings in Fig. 7 are merely examples, and the present invention is not limited to these settings. Also, it is not essential to set the focus drive amount in stages in the front-focus area. Returning to FIG. 1, the description of the configuration of the imaging device 1 will continue.

[0052] The pixel interpolation unit 17 receives an image in which phase difference detection pixels are missing pixels from the phase difference information reading unit 12, and generates an image in which the phase difference detection pixels are interpolated. 4, the image input to the pixel interpolation unit 17 is a missing image in which signals from phase difference detection pixels are missing. Therefore, the pixel interpolation unit 17 interpolates missing pixels (phase difference detection pixels) from surrounding pixels through interpolation processing. Note that a well-known general method may be used for the interpolation processing. For example, as shown in FIG. 4, when the pixel structure of the image sensor 11 is a Bayer array and G pixels are used as phase difference detection pixels, the pixel interpolation unit 17 calculates missing pixels by using a weighted average value of surrounding non-missing G pixels. The pixel interpolation unit 17 outputs the image in which the missing pixels have been interpolated to the image processing unit 18.

[0053] The image processing unit 18 generates an output image from the image in which missing pixels have been interpolated by the pixel interpolation unit 17. The image processing unit 18 is configured with a general process circuit. The image processing unit 18 generates, for example, images of each of the R, G, and B colors from an image having a pixel structure of a Bayer array. In this case, when generating an R image, for example, the image processing unit 18 generates the R image by interpolating pixels other than R in the Bayer array with surrounding R pixels. The same applies to the G and B images. The image processing unit 18 may also perform general image processing such as white balance adjustment, gain adjustment, and detail adjustment. The image processing unit 18 outputs the processed image as an output image of the imaging device 1. With the above-described configuration, the imaging device 1 can capture an image that is in focus with high precision.

[0054] [Operation of imaging device] Next, the operation of the imaging device 1 according to the embodiment of the present invention will be described with reference to FIG. 8 (and as appropriate, with reference to FIG. 1 for the configuration).

[0055] In step S1, the image sensor 11 captures an image of light incident through the lens 10. This image sensor 11 is an image sensor with phase difference detection pixels, and some of the pixels of a specific color (here, G) are designated as left-eye phase difference detection pixels and right-eye phase difference detection pixels, with the left and right halves shielded from light.

[0056] In step S2, the phase difference information readout unit 12 reads out pixel values ​​of different phase difference detection pixels from the images captured in step S1 to generate phase difference images (left-eye phase difference image L, right-eye phase difference image R). Note that here, the phase difference information readout unit 12 uses the attention area cutout unit 12a to cut out only the image of a predetermined attention area to generate the phase difference image. The phase difference image is processed in parallel from step S3 onwards, and the captured image in which the phase difference detection pixel is a missing pixel is processed in parallel from step S7 onwards.

[0057] In step S3, the sensitivity corrector 13 corrects the sensitivity difference between the phase difference images (left-eye phase difference image L, right-eye phase difference image R) generated in step S2. In step S4, the phase difference detection unit 14 detects the phase difference (detected phase difference P dif ) is detected by block matching or the like.

[0058] In step S5, the reference phase difference conversion unit 15 reads out the lens parameters from the lens driver 10a, and calculates a conversion coefficient corresponding to the lens parameters based on the detected phase difference P detected in step S4. dif By multiplying this, the reference phase difference Pref is calculated (see formula (1) above).

[0059] Here, the reference phase difference conversion unit 15 converts the detected phase difference corresponding to the lens parameters into a reference phase difference when the reference parameters (lens parameters serving as a reference) are used. In step S6, the focus control unit 16 controls the focus of the lens 10 based on the reference phase difference calculated in step S5. Here, the focus control unit 16 outputs the focus drive amount to the lens driver 10a by referring to a lookup table (LUT) in which a focus drive amount adjusted in advance to provide optimal focus control for the defocus amount determined from the reference phase difference when imaging is performed under the reference parameters.

[0060] On the other hand, in step S7, the pixel interpolation unit 17 interpolates the missing pixel of the captured image in which the phase difference detection pixel captured in step S2 is the missing pixel, using surrounding pixels. In step S8, the image processing unit 18 generates an output image from the image in which the missing pixels have been interpolated in step S7. Here, the image processing unit 18 performs general image processing such as generating an RGB image from the Bayer array image, performing white balance adjustment, and gain adjustment. This operation is repeatedly performed while imaging is being performed.

[0061] [Variations] Although the imaging device 1 according to the embodiment of the present invention has been described above, the present invention is not limited to this embodiment.

[0062] (Variation 1) Here, the image sensor 11 is a single-chip imaging type image sensor. However, a multi-chip image sensor such as a three-chip image sensor may be used as the image sensor 11. In this case, the incident light from the lens 10 is separated into each color light by a spectral prism or the like, and the image is captured by an image sensor corresponding to each color. For example, in the case of a three-chip image sensor, one or more of the RGB image sensors may be image sensors with phase difference detection pixels.

[0063] (Variation 2) Here, the phase difference detection pixels of the image sensor 11 are pixels having an aperture with the left or right half shielded from light. However, the openings are not limited to being on the left and right sides, but may be provided on the top and bottom.

[0064] (Variation 3) Here, the phase difference information reading section 12 detects a phase difference in the entire captured image or in one image of the image of the region of interest in the captured image cut out by the region of interest cutout section 12a. However, the phase difference information reading unit 12 may detect one or more subjects using a known object recognition technique, cut out the captured image for each subject region, and the phase difference detection unit 14 may detect the phase difference for each subject.

[0065] (Variation 4) Here, the imaging device 1 is configured to include a sensitivity correction unit 13. However, when a highly accurate image sensor 11 is used, this is not necessarily a necessary component and may be omitted. Of course, it is preferable to include the sensitivity correction unit 13 in order to further improve accuracy.

[0066] (Variation 5) Here, the imaging device 1 is configured to include the attention area cropping unit 12a inside the phase difference information readout unit 12, but the attention area cropping unit 12a may be configured to be separated from the phase difference information readout unit 12. In that case, the phase difference information readout unit 12 reads out pixel values ​​of phase difference detection pixels for the entire image from the image sensor 11 to generate a phase difference image, and the attention area cropping unit 12a crops out an image of the attention area from the phase difference image. [Explanation of symbols]

[0067] 1. Imaging device 10 Lenses 10a lens driver 11 Image sensor 12 Phase difference information readout unit 12a Region of interest extraction section 13 Sensitivity correction unit 14 Phase difference detection section 15 Reference phase difference conversion section 16 Focus control section 17 Pixel Interpolation Unit 18 Image processing section L G Left eye phase difference detection pixel R G Right-eye phase difference detection pixel H L ,H R opening

Claims

1. An imaging device that captures images through a focus-controllable lens, an image sensor in which two types of phase difference detection pixels having different aperture states are arranged at specific pixels; a phase difference information reading unit that reads out pixel values ​​of phase difference detection pixels having the same type of opening state from the image sensor and generates two phase difference images that serve as phase difference information; a phase difference detection unit that detects a phase difference between the two phase difference images; a reference phase difference conversion unit that converts the detected phase difference, which is the phase difference detected by the phase difference detection unit, into a reference phase difference, which is the phase difference when imaging is performed using the reference parameters, by a predetermined arithmetic expression using a ratio between a lens parameter at the time of imaging and a reference parameter, which is a reference lens parameter; a focus control unit that controls focus with a preset focus driving amount corresponding to the reference phase difference; An imaging device comprising:

2. The reference phase difference conversion unit converts the detected phase difference into P ｄｉｆ , the lens parameter at the time of image capture is Fn, the focal length is f, the focus distance is L, the reference parameter is Fn ｒｅｆ , the focal length is f ｒｅｆ , focus distance L ｒｅｆ When the above equation is satisfied, the reference phase difference P ｒｅｆ of, [Equation 1] 2. The imaging device according to claim 1, wherein the calculation is performed as follows:

3. 3. The imaging device according to claim 1, wherein the focus control unit controls the focus by referring to a lookup table in which the focus drive amount monotonically increases in accordance with an increase in the reference phase difference in an area where the focus position is further back than the subject, and in which the focus drive amount is set in stages with respect to the reference phase difference in an area where the focus position is closer to the subject.

4. 4. The imaging device according to claim 1, wherein the phase difference detection unit detects, as the phase difference, a relative position vector obtained by block matching the two phase difference images.

5. The imaging device according to claim 4 , wherein the phase difference detection unit performs block matching in arrangement directions of two different types of apertures of the phase difference detection pixels.

6. 6. The imaging device according to claim 1, further comprising: a region-of-interest cutout unit that cuts out a predetermined or externally specified region of interest from each of the two phase difference images and generates an image for detecting a phase difference by the phase difference detection unit.

7. 7. The imaging device according to claim 1, further comprising a sensitivity correction unit, provided in a stage preceding the phase difference detection unit, that corrects a sensitivity difference so that peak values ​​of sensitivity between images for which a phase difference is detected match.

8. 8. The imaging device according to claim 1, further comprising a pixel interpolation unit that generates an image by interpolating a pixel value of the phase difference detection pixel with pixel values ​​of pixels surrounding the phase difference detection pixel.

Images