Imaging device
The imaging device uses phase difference detection pixels with varied apertures in specific colors to correct sensitivity errors, enabling accurate focus shift calculations and improved autofocus and depth information capture.
Patent Information
- Application Number
- JP2020140748
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-08-24
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2040-08-24
AI Technical Summary
Conventional image plane phase difference detection methods for autofocus and depth information acquisition in imaging devices suffer from inaccuracies due to wavelength dependence of phase difference pixel sensitivity, making it difficult to accurately calculate focus shift amounts.
An imaging device with an image sensor that includes phase difference detection pixels of different aperture states arranged in specific colors, using block matching and correction factors to calculate focus shift based on a geometric model dependent on a specific color's wavelength, thereby correcting sensitivity differences.
Accurately calculates focus shift amounts independent of wavelengths other than the specific color, ensuring precise autofocus and depth information capture.
Smart Images

Figure 0007583551000004 
Figure 0007583551000005 
Figure 0007583551000006
Abstract
Description
[Technical field]
[0001] The present invention relates to an imaging device using an image plane phase difference detection technique. [Background technology]
[0002] In recent years, there has been an increasing need to add functions to imaging devices, such as autofocus functions and functions to obtain depth information simultaneously with images. To realize these functions, it is important to have technology to detect the focus state of an arbitrary subject from the captured video, and technology to accurately estimate distance information (amount of focus deviation) according to the difference between the lens focus position and the subject position from the detected focus state.
[0003] Image plane phase difference detection technology is well known as a method for detecting the focus state of a subject from a captured image. Image plane phase difference detection technology is a method for determining a relative distance vector on an image forming plane whose incident light sensitivity changes depending on the focus state, that is, a phase difference, by using an image sensor consisting of pixels (phase difference detection pixels) having incident light sensitivity corresponding to two or more exit pupils having different shapes (for example, see Patent Document 1).
[0004] The method described in Patent Document 1 is an example of an autofocus method using image plane phase difference detection technology. The method described in Patent Document 1 uses multiple image plane phase difference pixels that divide the exit pupil into regions, calculates a phase difference (defined as a defocus amount in the document) from the spatial shift amount of the image generated from each pixel, and determines the lens focus drive amount from the phase difference. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2016-99416 A Summary of the Invention [Problem to be solved by the invention]
[0006] However, Patent Document 1 does not clearly state a procedure for calculating the amount of focus deviation from the phase difference calculated from the image of the image plane phase difference pixel, or a procedure for determining the amount of lens focus drive. It is generally known that the phase difference obtained by the image plane phase difference detection technology contains an error due to the wavelength dependency of the incident light sensitivity of the phase difference pixel. Therefore, the conventional method has a problem in that it is difficult to detect the amount of focus deviation accurately.
[0007] The present invention has been made in consideration of the above-mentioned problems in the conventional art, and an object of the present invention is to provide an imaging apparatus capable of detecting an accurate amount of focus deviation. [Means for solving the problem]
[0008] In order to solve the above-mentioned problems, the imaging device of the present invention is an imaging device that captures images through a lens, and is configured to include an image sensor, a phase difference information readout unit, a phase difference detection unit, and a focus deviation calculation unit.
[0009] In such a configuration, the imaging device reads out pixel values of phase difference detection pixels having the same type of opening state from an image sensor in which two types of phase difference detection pixels having different opening states are arranged in pixels of a specific color by a phase difference information reading unit, and generates two phase difference images. These two phase difference images are images of the same subject captured with different openings, so that a phase difference occurs between them. Then, the imaging device detects the phase difference between the two phase difference images by block matching or the like using the phase difference detection unit.
[0010] The imaging device also calculates a focus error amount according to a specific color by the focus error amount calculation unit. biased talk Heart Correct By correction factor Phase difference Correction, From the corrected phase difference The amount of focus deviation is calculated. Note that the wavelength of a specific color is specified, and the color is opened according to the wavelength. biased talk The correction coefficient for correcting the heart rate can be obtained in advance by measurement.
[0011] In this way, the imaging device uses an image sensor in which two types of phase difference detection pixels with different opening states are arranged for pixels of a specific color, thereby achieving an opening that depends only on the wavelength of a specific color. biased talk This enables the imaging apparatus to specify the relationship between the phase difference and the amount of defocus as a geometric model, and to accurately calculate the amount of defocus from the phase difference. Effect of the Invention
[0012] The present invention provides the following excellent effects. According to the present invention, the amount of focus deviation can be calculated from the phase difference using a geometric model that depends only on the wavelength of a specific color, so that the amount of focus deviation can be calculated with high accuracy without being affected by the wavelengths of other colors. As a result, the present invention can capture an accurately focused subject image. [Brief description of the drawings]
[0013] [Figure 1] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Diagram 2] 2A and 2B are diagrams showing the configuration of an image sensor of the imaging device of FIG. 1, where (a) is a front view and (b) is a side view. [Diagram 3] 2A and 2B are structural diagrams of pixel units corresponding to phase difference detection pixels in the image sensor of the imaging device of FIG. 1, in which (a) shows a pixel structure having an opening in the left half, and (b) shows a pixel structure having an opening in the right half. [Figure 4] 11 is a diagram showing an example of a color structure of an imaging surface captured by an image sensor in which phase difference detection pixels are formed in a part of G pixels. FIG. [Diagram 5] FIG. 11 is an explanatory diagram for explaining a geometric model for calculating a focus deviation amount. [Figure 6] FIG. 11 is a graph showing the relationship between the incident angle of light and the sensitivity of a phase difference pixel. [Figure 7]4 is a schematic diagram showing the relationship between the focus position and the range of angles of incidence on the image sensor; FIG. [Figure 8] FIG. 11 is a graph showing an example of the relationship between the incident angle range and the correction coefficient obtained in advance by measurement. [Figure 9] 11 is a diagram showing an example of the configuration of a lookup table in which the correspondence between the incident angle range and the correction coefficient is divided into a plurality of regions of the image sensor. FIG. [Figure 10] 4 is a flowchart showing the operation of the imaging device according to the embodiment of the present invention. [Figure 11] FIG. 11 is a graph showing the relationship between the actual focus deviation amount and the calculated focus deviation amount, where (a) shows the case where a correction coefficient is not used, and (b) shows the case where a correction coefficient is used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] [Configuration of imaging device] First, the configuration of an imaging device 1 according to an embodiment of the present invention will be described with reference to FIG. The imaging device 1 captures an image via a lens 10. In particular, the imaging device 1 detects the amount of focus deviation and captures an image of a subject with high precision. As shown in FIG. 1, the imaging device 1 includes a lens 10, an image sensor 11, a phase difference information reading unit 12, a sensitivity correction unit 13, a phase difference detection unit 14, a focus deviation calculation unit 15, a focus control unit 16, a pixel interpolation unit 17, and an image processing unit 18.
[0015] The lens 10 forms an object image of a subject on the imaging surface of the image sensor 11. A general camera lens composed of one or more optical lenses can be used as the lens 10. The lens 10 may be configured to be separable from the imaging device 1. Here, the lens 10 is equipped with a lens driver 10a.
[0016] The lens driver 10a drives the focus position of the lens 10. The lens driver 10a moves the focus position of the lens 10 in the optical axis direction according to a driving amount output from a focus control unit 16, which will be described later.
[0017] The image sensor 11 has at least two types of phase difference detection pixels arranged in pixels of a specific color for performing image plane phase difference detection with different opening states. Here, the opening state refers to the position or shape of the opening, or both. The image sensor 11 converts light incident via the lens 10 into an image. Note that the following description will be given taking an example of a single-chip imaging type image sensor.
[0018] Now, the structure of the image sensor 11 will be described with reference to FIG. 2 and FIG. As shown in FIG. 2, the image sensor 11 includes a microlens array 11a, a color filter 11b, a wiring layer 11c having an opening, and a photodiode 11d.
[0019] The microlens array 11a is a lens group in which microlenses (convex lenses) are arranged two-dimensionally. Each microlens in the microlens array 11a corresponds to a pixel, and irradiates incident light onto a photodiode 11d arranged at the focal length of the microlens.
[0020] The color filter 11b has a narrow transmission spectrum that transmits light of a specific color wavelength. Here, the color filter 11b is an RG1G2B Bayer array filter with red (R), green (G), and blue (B) in a ratio of 1:2:1, but other colors and arrays may be used.
[0021] The wiring layer 11c is a layer on which wiring is formed for inputting and outputting voltage to and from the photodiode 11d, etc. In this wiring layer 11c, a transparent region where no wiring is formed is an effective pixel region, which guides incident light to the photodiode 11d.
[0022] The photodiode 11d converts incident light into an electrical signal. The incident surface of the photodiode 11d serves as an imaging surface for capturing an image. Here, the opening state of the wiring layer 11c is changed for a phase difference detection pixel corresponding to a predetermined position, and light to the photodiode 11d is partially blocked.
[0023] For example, as shown in FIG. 3, the wiring layer 11c blocks 50% of light on each side. S Aperture H L FIG. 3(b) shows an example in which the wiring layer 11c is formed so that light is irradiated by the right side R of the photodiode 11d. S Aperture H R In this example, the wiring layer 11c is formed so that light is irradiated by the wiring layer 11c. Note that, although a dedicated light shielding plate may be provided to form the light shielding region, it is more efficient in terms of structure to use the wiring layer 11c. In addition, the phase difference detection pixels are pixels of the same color on the imaging surface, for example, some or all of the G pixels. Of course, the R pixels and the B pixels may be phase difference detection pixels. In addition, although the opening position is changed in the horizontal direction here, the opening position may be changed in the vertical direction.
[0024] FIG. 4 shows an example of a color structure of an imaging surface captured by the image sensor 11, in which phase difference detection pixels are formed in part of G pixels on the imaging surface. FIG. 4 shows the phase difference detection pixels (left eye phase difference detection pixel L) that are arranged as part of the G (G2) pixels and shielded from incident light on the left and right. G and right-eye phase difference detection pixel R G ) are arranged alternately at predetermined intervals in two dimensions. In this way, the image sensor 11 generates a captured image made up of pixel values of the pixels of each color and the phase difference detection pixels from the incident light. Returning to FIG. 1, the description of the configuration of the imaging device 1 will continue. The image sensor 11 outputs the captured image to a phase difference information reading unit 12 .
[0025] The phase difference information reading unit 12 reads out the pixel values of the phase difference detection pixels having the same type of opening state from the image sensor 11, and generates two phase difference images that serve as phase difference information. Here, the phase difference information reading unit 12 includes a region of interest cutting unit 12a.
[0026] The region-of-interest clipping unit 12a clips a region of interest from each of the two phase difference images read by the phase difference information reading unit 12. The attention area is not particularly limited, but may be, for example, an area manually specified from the outside. In this case, the operator recognizes the output image of the imaging device 1 via a monitor (not shown), and specifies a subject or the like on which the operator particularly wants to focus by using a rectangular area or the like. Of course, this attention area may be specified using an existing machine-learned model such as a person or object detection model. Alternatively, the attention area may be a preset area such as the center of the captured image.
[0027] The attention area clipping unit 12a clips out the images of the phase difference detection pixels from the attention area of the captured image, and generates phase difference images (left eye phase difference image L, right eye phase difference image R). For example, when the image sensor 11 has the pixel structure shown in FIG. 4, the attention area cutout unit 12a cuts out the left-eye phase difference detection pixel L G The attention area cutout unit 12a cuts out an image consisting of only the right-eye phase difference detection pixel R as a left-eye phase difference image L. G An image consisting only of the right eye phase difference image R is extracted. The attention area cutout unit 12 a outputs the generated left eye phase difference image L and right eye phase difference image R to the sensitivity correction unit 13 . Moreover, the phase difference information reading unit 12 outputs to the pixel interpolation unit 17 an image captured by the image sensor 11 in which the phase difference detection pixels are missing pixels.
[0028] The sensitivity correction unit 13 corrects the sensitivity difference (brightness difference, contrast difference) between the phase difference images (left eye phase difference image L, right eye phase difference image R) generated by the phase difference information reading unit 12 (attention area clipping unit 12a).
[0029] For example, the sensitivity correction unit 13 normalizes each image so that the peak values (maximum value, minimum value) of the sensitivities of the left-eye phase difference image L and the right-eye phase difference image R match. Alternatively, the sensitivity correction unit 13 may correct the sensitivity difference between the left-eye phase difference image L and the right-eye phase difference image R using a general histogram matching method. Alternatively, the sensitivity correction unit 13 may correct the sensitivity difference between the left-eye phase difference image L and the right-eye phase difference image R by multiplying the images by a coefficient that matches the sensitivities obtained from the light blocking rate, pixel sensitivity, etc., that are designed or measured in advance of the phase difference detection pixels. The sensitivity correcting unit 13 outputs the corrected left-eye phase difference image L′ and right-eye phase difference image R′ to the phase difference detecting unit 14 .
[0030] The phase difference detection unit 14 detects the phase difference P between the left-eye phase difference image L′ and the right-eye phase difference image R′ after correcting the sensitivity. dif The phase difference P dif can be calculated as the magnitude of the relative position vector.
[0031] For example, the phase difference detection unit 14 obtains the magnitude of the relative position vector by performing block matching between the left-eye phase difference image L′ and the right-eye phase difference image R′. Note that the search direction of the blocks in the block matching is set to conform to the arrangement direction (light blocking direction) of the aperture. For example, in this case, the phase difference image is a left-eye phase difference detection pixel L′ as shown in FIG. G and the right-eye phase difference detection pixel R G Therefore, the phase difference detection unit 14 sets the search direction for block matching to the horizontal direction.
[0032] Furthermore, when performing block matching, the phase difference detection unit 14 uses sub-pixel units, which are smaller than pixels, as search units. This allows the phase difference detection unit 14 to detect phase differences more finely and accurately than the pixel units of the image sensor 11. The phase difference detection unit 14 detects the phase difference P dif to the focus deviation calculation unit 15.
[0033] The focus deviation calculation unit 15 calculates the phase difference P detected by the phase difference detection unit 14 using a correction coefficient that corrects the aperture decentering according to a specific color. dif is corrected, and the focus deviation amount is calculated from the corrected phase difference. The focus deviation calculation unit 15 calculates the focus deviation amount using a geometric model that represents the deviation amount between the lens focus distance and the distance to the actual subject, a correction coefficient that is defined by the incident light sensitivity of the phase difference detection pixel, and the phase difference.
[0034] Here, a method for calculating the amount of focus deviation in the focus deviation calculation section 15 will be described with reference to FIG. 5 is a diagram showing a geometric model that shows a schematic shift amount between the lens focus distance and the distance to the actual subject (subject distance) when a phase difference pixel with the lower half of the paper surface being light-shielded is used. Here, the lens 10 shows the optical system as a schematic convex lens. The defocus amount calculation unit 15 calculates the defocus amount X using the geometric model of FIG.
[0035]
number
[0036] The geometric model in Fig. 5 is a model showing the optical path of a light ray of a subject O that passes through the aperture decenter of the exit pupil (effective aperture determined by the diaphragm) of the lens 10 until the image height of the imaging surface of the image sensor 11 is "0", that is, until the image is formed at the center. Note that the phase difference pixels of the image sensor 11 are equivalent to half of the lens 10 being 50% light-shielded. Also, A denotes the ideal aperture decenter that is uniquely determined by the aperture shape of the lens 10 (aperture shape of the phase difference pixels), and C denotes a correction coefficient for correcting an error in the aperture decenter that occurs depending on the wavelength of the light. In other words, A x C denotes the aperture decenter after correction.
[0037] In the formulas (1) to (4), L D is the lens focus distance from the object O to the imaging surface of the image sensor 11 when in focus, and T D indicates the subject distance from the subject O' to the imaging surface of the image sensor 11 in an out-of-focus state. Also, L obj is the distance from the object O to the principal point of the lens 10, T obj indicates the distance from the object O' to the principal point of the lens 10. Also, L img is the distance from the principal point of the lens 10 to the imaging surface of the image sensor 11, T img indicates the distance from the principal point of the lens 10 to where the subject light reaches the optical axis of the lens 10. Furthermore, f represents the focal length of the lens 10.
[0038] Moreover, D in the above formula (3) is the imaging position of the image sensor 11 of the light beam that has passed through the aperture decenter of the lens 10 from the object O′. This D is the phase difference P detected by the phase difference detection unit 14. dif and the pixel pitch P of the phase difference pixel, using the above-mentioned formula (4). Note that the pixel pitch P of the phase difference pixel is known information that is set in advance.
[0039] In addition, in the aperture decentering A×C in the above formula (3), A is the center of gravity of the aperture area of the lens 10. In the example of this embodiment, since 50% of the light is blocked, ideally A=0.25×f / Fn However, the aperture decentering varies depending on the wavelength of the light. Therefore, here, a correction coefficient C is used to correct the error in the aperture decentering. This correction coefficient C is a pre-measured coefficient that corrects the error in the aperture decentering of a specific light (here, G light), and is a real number between 0 and 1. Note that, n The value is the ratio between the focal length of lens 10 and its exit pupil (effective aperture).
[0040] (Regarding correction coefficients) The correction coefficient C will be specifically described below. 6 is a graph showing the relationship between the incident angle [deg.] of light and the sensitivity [au] of the phase difference pixels when the R pixel, G pixel, and B pixel are used as phase difference detection pixels. As shown in FIG. 6, the sensitivity of the image sensor 11 varies depending on the incident angle and wavelength of light. However, since the imaging device 1 limits the phase difference detection pixels to pixels of a specific color (here, G pixels), it is only necessary to take into consideration the wavelength of the specific color.
[0041] The range of incident angles of light to the image sensor 11 is F n It is determined by the value (F-number) and focus position. Figure 7 shows the F n FIG. 1 is a schematic diagram showing the relationship between the focal length and focus position and the range of incident angles to the image sensor 11. Here, the range of incident angles θ can be calculated by the following formula (5). Note that the maximum range of incident angles θ when an object at infinity is photographed is max f = T img Therefore, we obtain the following equation (6).
[0042]
number
[0043] In an actual imaging system, f / T img is almost "1", and the value fluctuation is also arctangent (tan -1 ) is small enough relative to the variation of Fn It may be specified by the value alone. In this case, F n = 2.2, the incidence angle range θ max is approximately -12 to +12 (deg.). Then, based on the incident angle and sensitivity shown in FIG. 6, the sensitivity gravity center angle (θ R ,θ G ,θ B ) and the ideal sensitivity center of gravity angle (0.5 × θ), the correction coefficient (C R ,C G ,C B Here, it is assumed that the ideal sensitivity centroid angle is one in which the sensitivity of the phase difference pixel has a constant value on the light-sensitive side (positive incident angle range in the case of the left-eye phase difference detection pixel) and is “0” on the light-shielded side (negative incident angle range in the case of the left-eye phase difference detection pixel).
[0044]
number
[0045] FIG. 8 shows an example of the relationship between the incident angle range and the correction coefficient, which is obtained in advance by measurement. For example, F n = 2.2, the incident angle range is approximately ±12 degrees, so when using the R pixel as the phase difference detection pixel, the correction coefficient C R When using G pixels, the correction coefficient C G When using B pixel, the correction coefficient C B is set to “0.60”. This F n The correspondence between the incident angle ranges for each value and the correction coefficients may be stored in advance as a look-up table (LUT) in a memory (not shown) and the focus deviation calculation unit 15 may refer to it.
[0046] In addition, since the sensitivity to the incident angle differs between the center and the periphery of the image sensor 11, the image sensor 11 may be divided into a plurality of regions and a separate LUT may be provided for each region. For example, when the phase difference detection pixels having different light shielding between the left and right sides shown in Fig. 4 are used, a plurality of LUTs may be prepared by dividing the image sensor 11 into a plurality of left and right regions, such as LUT1 for the center of the image sensor 11 and LUT2, LUT3, and LUT4 for the left and right periphery thereof, as shown in Fig. 9.
[0047] Returning to FIG. 1, the description of the configuration of the imaging device 1 will continue. The focus deviation calculation unit 15 outputs the focus deviation X calculated by the above formula (1) to the focus control unit 16.
[0048] The focus control unit 16 controls the focus of the lens 10 based on the amount of focus deviation calculated by the focus deviation calculation unit 15 . The focus control unit 16 outputs a distance corresponding to the amount of focus deviation as a lens focus drive amount to the lens driver 10a, whereby the focus position of the lens 10 moves in a direction that eliminates the focus deviation.
[0049] The pixel interpolation unit 17 inputs an image in which phase difference detection pixels are missing pixels from the phase difference information reading unit 12, and interpolates the phase difference detection pixels. As described in Fig. 4, the image input to the pixel interpolation unit 17 is a missing image in which signals from phase difference detection pixels are missing. Therefore, the pixel interpolation unit 17 interpolates missing pixels (phase difference detection pixels) from surrounding pixels by an interpolation process. Note that a known general method may be used for the interpolation process. For example, as shown in FIG. 4, when the pixel structure of the image sensor 11 is a Bayer array and G pixels are used as phase difference detection pixels, the pixel interpolation unit 17 calculates missing pixels using a weighted average value of surrounding non-missing G pixels. The pixel interpolation unit 17 outputs the image in which the missing pixels have been interpolated to the image processing unit 18.
[0050] The image processing unit 18 generates an output image from the image in which missing pixels have been interpolated by the pixel interpolation unit 17. The image processing unit 18 is configured with a general process circuit. The image processing unit 18 generates images of each color of RGB from an image having a pixel structure of, for example, a Bayer array. In this case, when generating an R image, for example, the image processing unit 18 generates the R image by interpolating pixels other than R in the Bayer array with surrounding R pixels. The same applies to the G image and the B image. Furthermore, the image processing unit 18 may perform general image processing such as white balance adjustment and gain adjustment. The image processing unit 18 outputs the image after the image processing as an output image of the imaging device 1.
[0051] With the above-described configuration, the imaging device 1 can detect the amount of focus deviation only from light of a specific wavelength in image plane phase difference detection. This allows the imaging device 1 to capture an image that is precisely in focus without being affected by wavelengths other than the specific color.
[0052] [Operation of imaging device] Next, the operation of the imaging device 1 according to the embodiment of the present invention will be described with reference to FIG. 10 (for the configuration, refer to FIG. 1 as appropriate).
[0053] In step S1, the image sensor 11 captures light incident via the lens 10. This image sensor 11 is an image sensor with phase difference detection pixels, and some of the pixels of a specific color (here, G) are designated as left-eye phase difference detection pixels and right-eye phase difference detection pixels, with the left and right halves being shaded.
[0054] In step S2, the phase difference information readout unit 12 reads out pixel values of different phase difference detection pixels from the images captured in step S1 to generate phase difference images (left eye phase difference image L, right eye phase difference image R). Note that here, the phase difference information readout unit 12 uses the attention area cutout unit 12a to cut out only an image of a previously set attention area to generate a phase difference image. The phase difference image is processed in parallel from step S3 onwards, and the captured image in which the phase difference detection pixel is a missing pixel is processed in parallel from step S7 onwards.
[0055] In step S3, the sensitivity corrector 13 corrects the sensitivity difference between the phase difference images (left-eye phase difference image L, right-eye phase difference image R) generated in step S2. In step S4, the phase difference detection unit 14 detects the phase difference P between the left-eye phase difference image L′ and the right-eye phase difference image R′ corrected in step S3. dif are detected by block matching or the like.
[0056] In step S5, the focus deviation calculation unit 15 calculates the phase difference P dif and the aperture decentering corrected with a correction coefficient according to the wavelength of the color (here, G) of the phase difference image, the focus deviation amount is calculated by the above-mentioned formula (1). In step S6, the focus control unit 16 controls the lens driver 10a to move the focus position of the lens 10 using a distance corresponding to the amount of focus deviation calculated in step S5 as the lens focus driving amount.
[0057] On the other hand, in step S7, the pixel interpolation unit 17 interpolates the missing pixel of the captured image in which the phase difference detection pixel captured in step S2 is the missing pixel, using surrounding pixels. In step S8, the image processing unit 18 generates an output image from the image in which the missing pixels have been interpolated in step S7. Here, the image processing unit 18 performs general image processing such as generating an RGB image from the image of the Bayer array, performing white balance adjustment, and gain adjustment. This operation is repeatedly executed during imaging.
[0058] [Evaluation Results] Next, the evaluation results of the detection performance of the focus deviation amount in the imaging device 1 will be described with reference to Fig. 11. Fig. 11 shows the measurement results of the focus deviation amount when a white point light source placed at the center of the optical axis and 2 m away from the image sensor 11 is imaged while changing the focus position of the lens 10. The horizontal axis of Fig. 11 is the actual focus deviation amount, and the vertical axis is the focus deviation amount calculated by the focus deviation amount calculation unit 15. Note that here, the measurement results are shown for each case where the phase detection pixels are R pixels, G pixels, and B pixels.
[0059] FIG. 11(a) shows a case where no correction is performed (correction coefficient C=1), and FIG. 11(b) shows a case where correction is performed using a correction coefficient according to the wavelength of the color of the phase detection pixel. R = 0.24, the phase detection pixel is the G pixel, C G = 0.36, when the phase detection pixel is pixel B, C B =0.60.
[0060] As shown in Fig. 11(a), there is an error between the actual focus deviation amount and the calculated focus deviation amount. The measurement results also show that the focus deviation amount is wavelength dependent. The longer the wavelength of the incident light, for example, the larger the error becomes when using G pixels rather than B pixels, and when using R pixels rather than G pixels as phase detection pixels. In contrast, as shown in FIG. 11B, by performing correction using a correction coefficient according to the wavelength of the color of the phase detection pixel, the error in the focus deviation amount is reduced and the focus deviation amount can be calculated with high accuracy.
[0061] [Variations] Although the imaging device 1 according to the embodiment of the present invention has been described above, the present invention is not limited to this embodiment.
[0062] (Variation 1) Here, the phase difference is detected from left-eye phase difference detection pixels and right-eye phase difference detection pixels of the same color, but the phase difference may be detected from phase difference detection pixels of different colors. For example, if the color of the left eye is R and the color of the right eye is B, the focus deviation calculation unit 15 calculates the phase difference (C R +C B ) / 2 correction factor.
[0063] (Variation 2) Here, the image sensor 11 is a single-chip imaging type image sensor. However, an image sensor of a multi-chip imaging method such as a three-chip imaging method may be used as the image sensor 11. In this case, the incident light from the lens 10 is separated into each color light by a spectral prism or the like, and an image is captured by an image sensor corresponding to each color. For example, in the case of a three-chip imaging method, any one of the RGB image sensors may be an image sensor with phase difference detection pixels.
[0064] (Variation 3) Here, the phase difference detection pixels of the image sensor 11 are pixels having an aperture with the left or right half shielded from light. However, the opening is not limited to being on the left or right side, but may be on the top or bottom. In this case, too, the correction coefficient may be obtained in advance by measurement.
[0065] (Variation 4) Here, the imaging device 1 calculates the amount of focus deviation from the phase difference, but the phase difference detection section 14 may output the detected phase difference as it is as depth information. Here, the phase difference information reading section 12 detects a phase difference in the entire captured image or in one image of the image of the attention region in the captured image cut out by the attention region cutting section 12a. However, the phase difference information reading unit 12 may use a known object recognition technique to detect one or more subjects, cut out the captured image for each subject area, and have the phase difference detection unit 14 detect the phase difference for each subject and output it as depth information. It should be noted that the operation mode of the imaging device 1 may be switched externally to determine whether it operates to calculate the amount of focus deviation and control the focus position of the lens 10, or to output the depths of a plurality of subjects.
[0066] (others) Here, the imaging device 1 includes a focus control unit 16 and automatically controls the focus of the lens 10. However, the amount of focus deviation calculated by the focus deviation amount calculation unit 15 may be displayed on a monitor (not shown) and an operator may manually control the focus of the lens 10 according to the amount of deviation. In this case, the imaging device 1 may be configured without including the lens driver 10a and the focus control unit 16.
[0067] Also, here, the imaging device 1 is configured to include a sensitivity correction unit 13. However, when a high-precision image sensor 11 is used, this is not necessarily required as a configuration and may be omitted. Of course, it is preferable to include the sensitivity correction unit 13 in order to further increase the precision.
[0068] In addition, here, the attention area cutting unit 12a is configured to be provided inside the phase difference information readout unit 12, but may be configured to be separated from the phase difference information readout unit 12. In that case, the phase difference information readout unit 12 reads out pixel values of phase difference detection pixels in the entire image from the image sensor 11 to generate a phase difference image, and the attention area cutting unit 12a cuts out an image of the attention area from the phase difference image. [Explanation of symbols]
[0069] 1. Imaging device 10 Lenses 10a Lens driver 11 Image Sensor 12 Phase difference information readout section 12a Region of interest extraction section 13 Sensitivity correction section 14 Phase difference detection section 15 Focus deviation calculation unit 16 Focus control section 17 Pixel Interpolation Section 18 Image processing section L G Left eye phase difference detection pixel R G Right eye phase difference detection pixel H L ,H R opening
Claims
1. An imaging device that captures an image through a lens, An image sensor in which two types of phase difference detection pixels having different opening states are arranged in pixels of a specific color; a phase difference information reading unit that reads out pixel values of phase difference detection pixels having the same type of opening state from the image sensor and generates two phase difference images serving as phase difference information; a phase difference detection unit that detects a phase difference between the two phase difference images; a focus deviation amount calculation unit that corrects a phase difference using a correction coefficient that corrects the aperture decentering according to the specific color and calculates a focus deviation amount from the corrected phase difference; An imaging device comprising:
2. 2 . The imaging device according to claim 1 , wherein the phase difference detection section detects, as the phase difference, a relative position vector obtained by block matching the two phase difference images.
3. The imaging device according to claim 2 , wherein the phase difference detection unit performs block matching in an arrangement direction of two different types of apertures of the phase difference detection pixels.
4. 4. The imaging device according to claim 1, further comprising a focus control unit that controls a focus position of the lens using a distance corresponding to the focus shift amount calculated by the focus shift amount calculation unit as a driving amount.
5. The imaging device according to claim 1 or 4, further comprising an attention area extraction unit that extracts a predetermined or externally specified attention area from each of the two phase difference images and generates an image for detecting a phase difference by the phase difference detection unit.
6. 6. The imaging device according to claim 1, further comprising a sensitivity correction unit, disposed in front of the phase difference detection unit, for correcting a sensitivity difference so that peak values of sensitivity between images for detecting the phase difference match.
7. 7. The imaging device according to claim 1, further comprising a pixel interpolation unit that interpolates a pixel value of the phase difference detection pixel with pixel values of pixels of the specific color surrounding the phase difference detection pixel.
Citation Information
Patent Citations
Imaging apparatus
JP2012141585A
Image processing apparatus, image pickup apparatus, and image processing program
JP2013229764A
Solid state image pickup device, electronic apparatus, lens control method and image pickup module
JP2014236411A
Imaging device, control method of imaging device, and program
JP2016028265A
Imaging apparatus
JP2016099416A
