Imaging device

Abstract

To provide an imaging apparatus capable of obtaining a high quality image using a depth expansion optical system.SOLUTION: An imaging apparatus (100) includes an imaging optical system (101) including a wavefront modulation element (101c) rotationally asymmetric with respect to an optical axis, and an imaging unit (102) that photoelectrically converts an optical image formed by the imaging optical system. The imaging unit obtains first image data corresponding to a first color by first imaging, and obtains second image data corresponding to a second color by second imaging. The second image data is captured by shifting the first image data by a non-integer pixel of the imaging unit, and the angle between a first direction in which a modulation transfer function of the imaging optical system is the largest and a second direction which is the shortest of the shift directions is 30 degrees to 60 degrees.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] Conventionally, a technology has been proposed for expanding the depth of field by combining an optical system including a wavefront modulation element that modulates the phase of transmitted light with image processing. Specifically, the wavefront modulation element distributes light beams regularly, making the optical transfer function (OTF) of the optical system substantially invariant to changes in the subject distance. Then, an image with a deep depth of field is generated by performing restoration processing on an image obtained via an image sensor or the like. In addition, since the image sensor discretely samples the image of the subject field, the relationship between the rotational positions of the wavefront modulation element and the image sensor is arbitrary.

[0003] Patent Document 1 discloses an imaging device equipped with an optical system that forms a rotationally asymmetric point spread function (PSF) with a top. In order to realize a depth-extending optical system with loose tolerances, this imaging device sets the relative mounting positions of the optical system and the imaging element so that the tip of the diagonal side from the top is aligned with the diagonal direction of the pixel array. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-136144 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, pixel-enhancing processing is a process for increasing the number of pixels of an image beyond the number of pixels of an imaging element. A known method of pixel-enhancing processing is to use a pixel-shifted image to achieve high pixel density. When performing pixel shifting to achieve high pixel density, the two-dimensional sampling pattern is determined based not only on the pixel arrangement of the imaging element but also on the shifting method. However, the imaging device disclosed in Patent Document 1 does not take into account the relationship between the pixel shifting method and the installation of the imaging element, and therefore cannot obtain an image with good image quality when performing pixel-enhancing processing using pixel shifting.

[0006] SUMMARY OF THE PRESENT EMBODIMENTS In view of the above, an object of the present invention is to provide an imaging device capable of obtaining high-quality images using a depth-expanding optical system. [Means for solving the problem]

[0007] According to one aspect of the present invention, there is provided an imaging device comprising: an imaging optical system including a wavefront modulation element that is rotationally asymmetric with respect to an optical axis; and an imaging section that photoelectrically converts an optical image formed by the imaging optical system, , No. First image data corresponding to the first color And, The second image data corresponding to the second color and one or more shifted image data including Obtaining the said One or more shifted image data teeth each , the first image data is subjected to a non-integer pixel amount of the imaging unit The optical image is shifted by the amount of shift shift made Imaging Acquired by , the modulation transfer function of the imaging optical system is the largest Direction The first direction and and the one or more shifted image data shift amount Most of Shift direction corresponding to small shift amount Second Direction and the angle between the first direction and the second direction is θ [°], 30≦θ≦60 Satisfy the condition .

[0008] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging device capable of obtaining a high-quality image by using a depth expanding optical system. [Brief description of the drawings]

[0010] [Figure 1] 5 is an explanatory diagram of an image restoration filter in each embodiment. FIG. [Diagram 2] 4A to 4C are explanatory diagrams (sectional views) of an image restoration filter in each embodiment. [Diagram 3] FIG. 4 is an explanatory diagram of a point spread function in each embodiment. [Figure 4] 4A to 4C are explanatory diagrams of an amplitude component MTF and a phase component PTF of an optical transfer function in each embodiment. [Diagram 5] 5A to 5C are explanatory diagrams of the MTF of an optical system including a wavefront modulation element in each embodiment. [Figure 6] 11A to 11C are diagrams illustrating an example of imaging using pixel shifting and high pixel processing in each embodiment. [Figure 7] 1A to 1C are diagrams illustrating different examples of imaging using pixel shifting and pixel-enhancing processing in each embodiment. [Figure 8] 5A to 5C are explanatory diagrams of pixel-enhancing processing in each embodiment. [Figure 9] 10A to 10C are explanatory diagrams of Nyquist frequencies obtained by imaging using pixel shifting and high pixel processing in each embodiment. [Figure 10] FIG. 2 is a block diagram of an imaging device in each embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0012] First, before concretely describing the examples, the gist of the present invention will be described. The imaging device of this embodiment uses a depth-expanding optical system using a wavefront modulation element that is rotationally asymmetric with respect to the optical axis as the imaging optical system. A plurality of captured images corresponding to a plurality of colors are acquired, and pixel-high resolution processing is performed by pixel shifting. In addition, sharpening processing (image restoration processing) is performed based on the optical characteristics of the optical system. Below, various technologies used in this embodiment will be described before describing the configuration of the imaging device of this embodiment.

[0013] [Image recovery processing] First, an overview of the image restoration process will be described. When the captured image (degraded image) is g(x,y), the original image is f(x,y), and the point spread function PSF, which is a Fourier pair of the optical transfer function OTF, is h(x,y), the following formula (1) is established.

[0014] g(x,y)=h(x,y)*f(x,y) … (1) Here, * denotes convolution (convolution integral, product sum), and (x, y) are coordinates on the captured image.

[0015] Furthermore, if equation (1) is Fourier transformed and converted into a frequency domain display format, equation (2), expressed as a product for each frequency, is obtained.

[0016] G(u,v)=H(u,v)·F(u,v) … (2) Here, H is the optical transfer function OTF obtained by Fourier transforming the point spread function PSF(h), and G and F are functions obtained by Fourier transforming the degraded image g and the original image f, respectively. (u,v) are the coordinates in the two-dimensional frequency plane, i.e., the frequency.

[0017] To obtain the original image f from the captured degraded image g, it is sufficient to divide both sides of the following equation (3) by the optical transfer function H.

[0018] G(u,v) / H(u,v)=F(u,v) … (3) Then, F(u,v), i.e., G(u,v) / H(u,v), is transformed back to the real plane by inverse Fourier transform, and the original image f(x,y) is obtained as a restored image.

[0019] H -1 If R is the inverse Fourier transform of, then the original image f(x, y) can be obtained by performing convolution processing on the image on the real plane as shown in the following equation (4).

[0020] g(x,y)*R(x,y)=f(x,y) … (4) Here, R(x,y) is called an image restoration filter. When the image is a two-dimensional image, the image restoration filter R is generally a two-dimensional filter having taps (cells) corresponding to each pixel of the image. In addition, the more taps (cells) the image restoration filter R has, the higher the restoration accuracy is. For this reason, a feasible number of taps is set according to the required image quality, image processing capability, aberration characteristics, and the like. The image restoration filter R must at least reflect the characteristics of aberration, and therefore is different from conventional edge enhancement filters with about three taps each for the horizontal and vertical directions. The image restoration filter R is set based on the optical transfer function OTF, and therefore can correct both the deterioration of the amplitude component and the phase component with high accuracy.

[0021] Furthermore, since actual images contain noise components, if the image restoration filter R created by taking the reciprocal of the optical transfer function OTF as described above is used, the noise components will be significantly amplified while restoring the degraded image. This is because the MTF (amplitude component) of the optical system is raised to return it to 1 across all frequencies in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is the amplitude degradation caused by the optical system, returns to 1, but at the same time the power spectrum of the noise is also raised, and as a result, the noise is amplified according to the degree to which the MTF is raised (restoration gain).

[0022] Therefore, when noise is included, an image of good quality for viewing cannot be obtained. This is expressed by the following equations (5-1) and (5-2). G(u,v)=H(u,v)·F(u,v)+N(u,v)…(5-1) G(u,v) / H(u,v)=F(u,v)+N(u,v) / H(u,v)…(5-2) Here, N is the noise component.

[0023] For images containing noise components, there is a method for controlling the degree of restoration according to the intensity ratio SNR of an image signal to a noise signal, such as a Wiener filter expressed by the following equation (6).

[0024]

number

[0025] Here, M(u,v) is the frequency characteristic of the Wiener filter, and |H(u,v)| is the absolute value of the optical transfer function OTF (modulation transfer function MTF). In this method, for each frequency, the smaller the MTF is, the smaller the restoration gain (degree of restoration) is, and the larger the MTF is, the larger the restoration gain is. Generally, the MTF of an imaging optical system is high on the low frequency side and low on the high frequency side, so this method essentially reduces the restoration gain on the high frequency side of the image.

[0026] Next, the image restoration filter will be described with reference to Fig. 1 and Fig. 2. Fig. 1 and Fig. 2 are explanatory diagrams of the image restoration filter. The number of taps of the image restoration filter is determined according to the aberration characteristics of the imaging optical system and the required restoration accuracy. The image restoration filter of Fig. 1 is, as an example, a two-dimensional filter with 11 × 11 taps. Although the values ​​(coefficients) in each tap are omitted in Fig. 1, a cross section of this image restoration filter is shown in Fig. 2. The distribution of the values ​​(coefficient values) of each tap of the image restoration filter has the function of ideally returning the signal value (PSF) that has been spatially spread due to aberration to the original single point.

[0027] Each tap of the image restoration filter is subjected to convolution processing (convolution integral, product-sum) in the image restoration process corresponding to each pixel of the image. In the convolution processing, in order to improve the signal value of a specific pixel, the pixel is aligned with the center of the image restoration filter. Then, for each corresponding pixel of the image and the image restoration filter, the product of the image signal value and the filter coefficient value is calculated, and the sum of these products is replaced as the signal value of the center pixel.

[0028] Next, the characteristics of image restoration in real space and frequency space will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is an explanatory diagram of the point spread function PSF, where Fig. 3(a) shows the point spread function PSF before image restoration, and Fig. 3(b) shows the point spread function PSF after image restoration. Fig. 4 is an explanatory diagram of the amplitude component MTF (Fig. 4(M)) and phase component PTF (Fig. 4(P)) of the optical transfer function OTF. The dashed line (a) in Fig. 4(M) shows the MTF before image restoration, and the dashed line (b) shows the MTF after image restoration. The dashed line (a) in Fig. 4(P) shows the PTF before image restoration, and the dashed line (b) shows the PTF after image restoration. As shown in Fig. 3(a), the point spread function PSF before image restoration has an asymmetric spread, and due to this asymmetry, the phase component PTF has a nonlinear value with respect to frequency. In the image restoration process, the amplitude component MTF is amplified and the phase component PTF is corrected to zero, so that the point spread function PSF after image restoration has a symmetric and sharp shape. In this way, the image restoration filter can be obtained by performing an inverse Fourier transform on a function designed based on the inverse function of the optical transfer function OTF of the imaging optical system. The image restoration filter that can be used as the sharpening filter in each embodiment can be changed as appropriate, and for example, the Wiener filter described above can be used. When using a Wiener filter, it is possible to create an image restoration filter in real space that is actually convoluted with an image by performing an inverse Fourier transform on Equation (6).

[0029] [Extended depth of field] In each embodiment, a wavefront modulation element (phase mask) that reduces the performance variation in the depth direction is inserted into the optical system, and a method (WFC: WaveFront Coding) is performed to expand the depth. The wavefront modulation element provides a phase with a two-dimensional distribution in a plane perpendicular to the optical axis of the optical system, and may be, for example, an optical element whose thickness or refractive index changes in the plane or a liquid crystal spatial phase modulation element, and may be provided as an aspheric shape to an optical element in the optical system. In each embodiment, a cubic phase mask whose surface shape is expressed by a cubic function is used. If the direction perpendicular to the imaging element (the optical axis direction of the optical system) is the Z axis, and two axes perpendicular to the Z axis and perpendicular to each other are the X axis and the Y axis, the phase distribution φ of the wavefront modulation element in each embodiment can be expressed by an odd function shape as shown in the following formula (7).

[0030] φ=α(X 3 +Y 3 ) … (7) Here, X and Y do not necessarily coincide with the x and y of the captured image or the imaging element, and hereinafter, the axial direction of the wavefront modulation element (the first axis and the second axis perpendicular to each other) is represented by X and Y, respectively. By providing this phase distribution, the performance variation in the depth direction can be blunted. In this embodiment, an optical system in which the depth is expanded using the wavefront modulation element is called a depth expansion optical system. The phase distribution used for the depth expansion is not limited to the above. Since the robustness against the change in defocus and the balance between the high frequency side and the low frequency side of the MTF change, it may be appropriately selected according to the required depth of field and frequency characteristics. By reducing the variation in the optical performance in the depth direction by the wavefront modulation element, the image restoration process can be applied using the same image restoration filter regardless of the depth. In addition, when a phase mask is inserted into the optical system to blunt the performance variation in the depth direction, the performance variation also tends to be blunted with respect to the shooting conditions such as the image height and the object distance. In other words, if the performance variation with respect to image height in a depth-of-field optical system is sufficiently small, it can be treated as a uniform OTF for shooting conditions such as image height and object distance, and a uniform (fixed) image restoration filter can be used.

[0031] FIG. 5 is a cross section of the MTF of an optical system including a wavefront modulation element expressed by formula (7). Here, the aberration of the optical system is assumed to be small compared to the aberration caused by the wavefront modulation element. In FIG. 5, the horizontal axis represents the spatial frequency, F represents the F-number (aperture value) of the optical system, the solid line represents the MTF in the X-axis direction, and the dashed line represents the MTF in the diagonal direction (diagonal 45 degrees). When the wavefront modulation element expressed by formula (7) is used, the X-axis direction and the Y-axis direction are symmetrical, so the MTF for the spatial frequency in the X-axis direction and the Y-axis direction match. However, when comparing the MTF of the solid line with the MTF of the dashed line, the MTF of the dashed line is low from the low frequency side to the high frequency side, and there is a large difference from the MTF of the solid line. Thus, although it is generally believed that the depth can be expanded by using a cubic phase mask, the MTF characteristics differ greatly depending on the direction.

[0032] [Pixel shift imaging and high pixel processing] Next, an overview of the pixel-enhancing process will be described. In the pixel-enhancing process, a pixel-enhanced image is obtained from a plurality of captured images taken at different positions. In each embodiment, image data including images corresponding to a plurality of colors is obtained using a monochromatic image sensor in which all pixels receive light corresponding to the same wavelength range. At least one image included in the image data is captured by shifting it by a non-integer pixel with respect to a reference image included in the image data. Here, the "reference image" is an arbitrary image included in the image data that can be selected as a reference, and is a reference for defining the amount and direction of the shift. Also, "imaging with a shift" refers to capturing an image by translating the position where the image of the subject formed by the imaging optical system is sampled in a plane perpendicular to the optical axis.

[0033] For example, by separating light passing through the imaging optical system into colors using a prism and arranging multiple imaging elements corresponding to each separated light beam, images corresponding to multiple colors can be simultaneously obtained. Since each imaging element corresponds to a different color, it may have sensitivity to a different wavelength. For example, it can be separated into three light beams of RGB and acquired. The number of separations is not limited to three, and may be two or four or more. In addition, in the case of three or more, multiple images corresponding to the same color may be acquired. Then, by arranging one imaging element by shifting it by a non-integer pixel relative to another imaging element, it is possible to sample the image of the subject at a position shifted by a non-integer pixel. Also, "arranging the imaging element by shifting" refers to arranging the imaging element so that the position where the image of the subject formed by the imaging optical system is sampled is moved in parallel in a plane perpendicular to the optical axis and imaged. Therefore, when the light beam is separated and bent, it is sufficient to shift it in a plane perpendicular to the optical axis.

[0034] Also, imaging may be performed multiple times with one image sensor. Assuming that there is no external light, the color of the illumination light illuminating the subject is changed in synchronization with the imaging, so that the images acquired by imaging multiple times correspond to different colors based on the illumination light. By imaging while driving the image sensor in a direction perpendicular to the optical axis, images corresponding to multiple colors are relatively shifted. For example, such a method may be used in an endoscope device or the like, since there is no external light. Instead of driving the image sensor, the optical system, some lenses included in the optical system, or the entire image sensor may be moved.

[0035] A more preferable method is to separate the light that has passed through the imaging optical system by color and arrange multiple imaging elements corresponding to each separated light beam. This eliminates the need for driving, so multiple images that are shifted by non-integer pixels can be acquired in one capture. This makes it possible to suppress the effects of subject blurring and capture the images as a video. The high pixel processing described below can be performed for each frame of the video in the same way as for still images. By performing high pixel processing on images captured using this imaging method, it is possible to obtain high pixel images with a resolution that exceeds that which can be obtained with a single imaging element.

[0036] Next, referring to FIG. 6, an example of imaging and high pixel processing by pixel shifting will be shown. FIG. 6(A) is an example of an imaging pattern shifted relatively by a non-integer pixel. Here, three images, a reference image 51, an image 52, and an image 53, are acquired. The reference image 51 and the image 52 are captured at the same position without shift. On the other hand, the image 53 is captured with a half pixel shift in the horizontal and vertical directions with respect to the reference image 51 and the image 52. For example, when capturing images using different image sensors for RGB, the capturing position of G can be shifted by a half pixel with respect to the capturing positions of R and B. That is, the reference image 51, the image 52, and the image 53 correspond to R, B, and G, respectively. Also, as shown in FIG. 6(B), an image of G that is not shifted relatively to the reference image 51 may be acquired as an image 54. Based on the multiple images included in the acquired image data, a high pixel processing to be described later is performed, and a high pixel image 55 having a larger number of pixels than each captured image can be acquired.

[0037] Next, referring to FIG. 7, a different example of imaging by pixel shifting and high pixel processing will be shown. FIG. 7 is an example of imaging in which the imaging pattern is different from that of FIG. 6 and the imaging is relatively shifted by a non-integer pixel. Here, four captured images to be used in high pixel processing are obtained by performing imaging with the shooting positions shifted from each other by half a pixel. The four captured images are a reference image 61 (without shift), an image 62 shifted by half a pixel in the horizontal direction, an image 63 shifted by half a pixel in the diagonal direction, and an image 64 shifted in the vertical direction. The image 63 is equivalent to a position shifted by half a pixel in both the horizontal and vertical directions. By obtaining the four captured images, a high pixel image 65 with four times the number of pixels can be obtained by high pixel processing. For example, when imaging is performed using different imaging elements corresponding to four colors, the shooting positions can be shifted to obtain the images, and the reference image 61, image 62, image 63, and image 64 correspond to R, G, B, and Y (yellow), respectively. Also, the reference images 61 and 63 may be G, the image 62 may be R, and the image 64 may be B.

[0038] In the high pixel processing shown in FIG. 7, the high pixel processing can be performed simply by arranging four images according to the sampling positions of the image of the subject. The high pixel processing in FIG. 6(A) is performed based on image 51 (or image 52) and image 53. FIG. 8 is an explanatory diagram of the high pixel processing. The white circles shown by solid lines and the white circles shown by dashed lines in FIG. 8 are the sampling positions of image 51 and image 53, respectively. The sampling positions may be, for example, the center positions of the pixels. The luminance value of the image corresponding to the position of the black circle can be obtained by interpolating the white circles shown by solid lines and the white circles shown by dashed lines with the nearest neighbors. The obtained black circle is twice as pixelated in the horizontal and vertical directions as the image 51 and the image 53, and the luminance values ​​corresponding to the positions of the black circles are arranged to form a high pixel processing image 55. As described above, the high pixel processing in each embodiment is performed based on a plurality of relatively shifted images, so that the effective sampling of the image of the subject is increased. That is, unlike high pixel count processing by interpolation from a single image, a high pixel count image is generated that has a resolution up to a higher frequency than the Nyquist frequency corresponding to the single image or the Nyquist frequency of the image sensor. The Nyquist frequency will be described in detail later.

[0039] However, in each embodiment, a pixel-enhancing process is performed to obtain a high-pixel image of a predetermined color based on images of different colors shifted by a non-integer pixel. Since the pixel-enhancing process is based on images corresponding to different colors, it differs from the case of capturing images with a smaller pixel pitch (sampling period) and high pixel count image sensor. Therefore, the difference between images corresponding to different colors is reduced, and a high-pixel image of the predetermined color is generated based on an image of the predetermined color and an image of a color different from the predetermined color, based on the pixel-enhancing process described above. When the saturation of the subject is low and close to black and white, the difference between images corresponding to different colors can be reduced by taking a white balance.

[0040] On the other hand, in a normal color subject, the ratio of colors is different, so by acquiring only the change in luminance value in the image of each color, the difference between images corresponding to different colors can be reduced. As a result, by acquiring only the change in luminance value in the image of each color and interpolating between different colors, it is possible to sample luminance changes up to high frequencies corresponding to a highly pixelated image. Since the highly pixelated process can be performed on colors that are relatively shifted by a non-integer pixel, it is possible to perform the highly pixelated process corresponding to all colors. For example, by performing the highly pixelated process on R and G, highly pixelated images of R and G can be obtained, and by performing the highly pixelated process on B and G, a highly pixelated image of B can be obtained. In addition, the RGB image may be converted into luminance and color difference signals, and the highly pixelated process may be performed on the luminance signal. Note that the highly pixelated process and the shift pattern of the captured image used therefor are not limited to those described above, and other methods may be used.

[0041] [Nyquist frequency] Next, the relationship between image sampling and the Nyquist frequency will be described with reference to FIG. 9. FIG. 9 is an explanatory diagram of the Nyquist frequency resulting from imaging using pixel shifting and high pixel processing. When an image of a subject is sampled at pixel pitch p, i.e., sampling period p, the sampling frequency is the reciprocal of the sampling period, 1 / p. The Nyquist frequency is half the sampling frequency, i.e., 1 / 2p. According to the sampling theorem, when an object is sampled, it can be accurately reproduced as long as the object is at or below the Nyquist frequency. Since an image is a two-dimensional sampling of an image of a subject, the Nyquist frequency is also given in two dimensions.

[0042] The Nyquist frequency when sampling is performed using the white circles shown by the solid lines in FIG. 9A is the dashed line 81 in FIG. 9B. Even when sampling is performed using the white circles shown by the dashed lines in FIG. 9A, the sampling positions are shifted, but the period of the two-dimensional arrangement and the arrangement direction do not change. Therefore, the Nyquist frequency when sampling is performed using the white circles shown by the dashed lines is similarly the dashed line 81 in FIG. 9B. However, the Nyquist frequency when sampling is performed using both the white circles shown by the solid lines in FIG. 9A and the white circles shown by the dashed lines in FIG. 9A is the solid line 82 in FIG. 9B. This corresponds to the Nyquist frequency in a highly pixelated image when a highly pixelated process corresponding to FIG. 6 is performed. On the other hand, the white circles shown by the solid lines in FIG. 9C are the same as those in FIG. 9A, but the white circles shown by the dashed lines correspond to FIG. 7 and are more numerous than those in FIG. 9A. The Nyquist frequency when sampling is performed at all of the white circles indicated by the solid line and the white circles indicated by the dashed line in FIG. 9C is shown by a solid line 83 in FIG. 9D.

[0043] Therefore, when the pixel pitch of the imaging element used in each embodiment is p, the Nyquist frequency of each captured image is the dashed line 81 in FIG. 9(B). However, as described above, when a high-pixel image of a predetermined color is generated based on images of different colors, the Nyquist frequency changes. Therefore, the Nyquist frequency is not necessarily the same for each image included in the image data and the high-pixel image. For example, in the example of FIG. 9(A) and (B), the direction in which the Nyquist frequency is highest also changes from the diagonal direction to the x direction and y direction. The image restoration process, the depth of field extension, pixel shift imaging and high-pixel processing, and the two-dimensional Nyquist frequency in this embodiment have been described above.

[0044] Next, the configuration of the imaging device in each embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram of the imaging device 100. An image processing program for performing the sharpening process (image restoration process) of this embodiment is installed in the imaging device 100, and this sharpening process is executed by an image processing unit (image processing device) 104 inside the imaging device 100.

[0045] The imaging device 100 includes an optical system (imaging optical system) 101 and an imaging device body (camera body). The optical system 101 includes an aperture 101a, a focus lens 101b, and a wavefront modulation element 101c, and is configured integrally with the camera body. However, each embodiment is not limited to this, and can also be applied to an imaging device in which the optical system 101 is detachably attached to the camera body. Furthermore, the optical system 101 may be configured to include an optical element having a refractive surface such as a lens, an optical element having a diffractive surface, and an optical element having a reflective surface.

[0046] The wavefront modulation element 101c is non-rotationally symmetric with respect to the optical axis. The wavefront modulation element 101c is disposed adjacent to the image side of the aperture 101a, but is not limited thereto, and may be disposed at a position different from that shown in Fig. 10. The imaging optical system is composed of optical elements disposed on the object side of the imaging unit 102.

[0047] The imaging unit 102 has one or more imaging elements such as a CCD or CMOS sensor, and generates a captured image by photoelectrically converting the subject image formed by the optical system 101 (the optical image formed by the optical system 101). That is, the subject image is converted into an analog signal (electrical signal) by photoelectric conversion by the imaging unit 102. Then, this analog signal is converted into a digital signal by the A / D converter 103. This digital signal is input to the image processing unit 104.

[0048] The image processing unit 104 performs a predetermined process on the digital signal, and also performs the sharpening process of this embodiment. In addition to the predetermined process, the image processing unit 104 also performs pixel-enhancing process and image restoration process.

[0049] Optical characteristic data such as OTF and PSF required for the sharpening process, restoration filters, or data required for generating them are held in a storage unit (storage means) 108. The storage unit 108 is configured with, for example, a ROM. An output image processed by the image processing unit 104 is stored in a predetermined format in an image recording medium 109. An image obtained by performing predetermined processing for display on an image that has been subjected to sharpening processing is displayed on a display unit 105 configured with a liquid crystal monitor or an organic EL display. However, the image displayed on the display unit 105 is not limited to this, and an image that has been subjected to simple processing for high-speed display may be displayed on the display unit 105.

[0050] The system controller 110 controls the imaging device 100. The optical system 101 is mechanically driven by the optical system control unit 106 based on instructions from the system controller 110. The optical system control unit 106 controls the aperture diameter of the diaphragm 101a to obtain a predetermined F-number. The optical system control unit 106 also controls the position of the focus lens 101b by an autofocus (AF) mechanism or a manual focus mechanism (not shown) to adjust the focus according to the subject distance. Note that functions such as the aperture diameter control of the diaphragm 101a and manual focus may not be executed depending on the specifications of the imaging device 100.

[0051] Note that optical elements such as a low-pass filter and an infrared cut filter may be disposed between the optical system 101 and the imaging unit 102, but when an element that affects the optical characteristics such as a low-pass filter is used, consideration may be required when creating the sharpening filter. The infrared cut filter also affects each PSF of the RGB channels, which is the integral value of the point spread function (PSF) of the spectral wavelength, particularly the PSF of the R channel, so consideration may be required when creating the sharpening filter. Therefore, the sharpening filter may be changed depending on the presence or absence of a low-pass filter or an infrared cut filter.

[0052] The image processing unit 104 is configured with an ASIC, and the optical system control unit 106, the state detection unit 107, and the system controller 110 are each configured with a CPU or an MPU. One or more of the image processing unit 104, the optical system control unit 106, the state detection unit 107, and the system controller 110 may be configured with the same CPU or MPU.

[0053] [Example 1] Next, a first embodiment of the present invention will be described. In this embodiment, light passing through an imaging optical system is separated into three light beams corresponding to RGB by a prism, and imaging elements corresponding to each of them are arranged. All of the imaging elements have a square lattice pixel arrangement with the same pixel pitch, and the rotation directions around the optical axis are the same, that is, the pixel arrangement directions of the three imaging elements are the same. Here, the "direction of pixel arrangement" refers to the direction in which the interval between pixel centers is narrowest, and in the case of a square lattice pixel arrangement, it is the horizontal and vertical directions. Therefore, in this embodiment, the two-dimensional Nyquist frequency is the same for all of RGB. Then, only the imaging element corresponding to G is arranged by shifting it by a non-integer pixel relative to the imaging elements corresponding to R and B. Here, it is shifted in the diagonal direction by shifting it by half a pixel in the horizontal and vertical directions. At this time, the shift pattern during imaging corresponds to FIG. 6(A). Image data consisting of three images corresponding to RGB, respectively, is acquired by these three imaging elements. In addition, the wavefront modulation element has a phase distribution according to formula (7). The X and Y directions of the wavefront modulation element coincide with the pixel array directions (x and y directions).

[0054] Here, the Nyquist frequency of each of RGB corresponds to the broken line 81 in FIG. 9B. In this case, the direction of the highest Nyquist frequency corresponding to each color of RGB is the diagonal direction in the lattice of the pixel array of the image sensor. On the other hand, the direction of the highest MTF of the image sensor is determined by the direction around the optical axis of the wavefront modulation element, and is the X direction and the Y direction in formula (7). In order to make the most of the performance of the image sensor, it is preferable that the direction of the highest MTF of the image sensor and the direction of the highest Nyquist frequency of the image coincide with each other. However, in this embodiment, pixel-enhancing processing is performed on all colors of RGB, and the Nyquist frequency of each of RGB in the pixel-enhancing image corresponds to the solid line 82 in FIG. 9B. That is, the Nyquist frequency of the image (the Nyquist frequency in each image) is the highest in the diagonal direction, but the direction of the highest Nyquist frequency changes before and after image processing by performing pixel-enhancing processing using images of different colors that are shifted in the diagonal direction. Therefore, in the arrangement of the image sensor and the high pixelation process of this embodiment, by arranging the pixels so that their array directions (x direction and y direction) coincide with the X direction and Y direction of the wavefront modulation element, the performance of the image sensor can be maximized to obtain a high pixelation image with a higher sense of resolution.

[0055] In this embodiment, the RGB images are captured one by one, but as described in FIG. 6B, the arrangement may be such that the G image that is not shifted with respect to the R and B images can be captured. This can be realized by further separating the light beam corresponding to G and arranging the image pickup element for the two separated light beams. Therefore, the image data consists of four images RGGB. In this case, the Nyquist frequency of each image does not change, but sampling is performed for G at the positions of the white circle shown by the solid line in FIG. 9A and the white circle shown by the dashed line, and the Nyquist frequency corresponding to G is the solid line 82 in FIG. 9B. In this case, the directions with the highest Nyquist frequency differ between the R, B images and the G image included in the image data, and the optimal direction of the wavefront modulation element is not uniquely determined. However, by performing the above-mentioned pixelation processing, the direction with the highest Nyquist frequency coincides with the horizontal direction, so it is sufficient to arrange the pixel array direction and the X and Y directions of the wavefront modulation element to coincide with each other. If high pixel count processing is not performed by performing sampling shifted diagonally by G, the direction of the highest Nyquist frequency corresponding to each color will be different, and the orientation of the wavefront modulation element cannot be optimized for all colors in the image data. However, by combining different colors and performing high pixel count processing, the direction of the highest Nyquist frequency of the high pixel count image can be aligned for all colors. Therefore, the optimal arrangement is one in which the pixel array direction (x direction, y direction) and the X direction and Y direction of the wavefront modulation element are aligned.

[0056] Here, "arrangement in which the pixel array direction (x direction, y direction) and the X direction and Y direction of the wavefront modulation element are aligned" corresponds to an arrangement in which the wavefront modulation element is arranged so that the direction in which the MTF of the imaging optical system is the largest forms an angle of 45 degrees with respect to the shift direction, which is the diagonal direction of the imaging element. Note that "diagonal direction of the imaging element" means a diagonal direction in the lattice of the pixel array of the imaging element. That is, the angle between the direction in which the MTF of the imaging optical system is the largest (first direction) and the shortest direction of the shift directions (second direction) is approximately 45 degrees. By bringing the angle between the first direction and the second direction closer to 45 degrees, the performance of the imaging optical system can be utilized, but it is not strictly limited to 45 degrees, and it is sufficient that it is approximately 45 degrees. For example, this angle is set to 30 degrees or more and 60 degrees or less. More preferably, this angle is set to 40 degrees or more and 50 degrees or less. The direction (first direction) in which the MTF of the imaging optical system is largest is the two directions, the X direction and the Y direction of the wavefront modulation element (the directions indicated by arrows a and b in FIGS. 9A and 9C), each of which is at an angle of 45 degrees to the shift direction (the second direction (the direction indicated by arrow c)).

[0057] According to this embodiment, by taking into consideration the shift caused by pixel shifting during imaging using the depth expansion optical system, it is possible to realize an imaging device capable of capturing images with more preferable image quality.

[0058] [Example 2] Next, a second embodiment of the present invention will be described. In this embodiment, the pattern of shifting the image pickup element is different from that of the first embodiment. The light passing through the imaging optical system is separated into four light beams corresponding to RGBY (Y is yellow) by a prism, and the corresponding image pickup element is arranged for each. All four image pickup elements have the same pixel pitch and a square lattice pixel arrangement, and the rotation direction around the optical axis is the same, that is, the direction of the pixel arrangement is the same. Therefore, in this embodiment, the two-dimensional Nyquist frequency is the same for all RGBY. ​​The image pickup elements are arranged so that the image obtained by the R image pickup element is a reference image, the image corresponding to G is shifted by half a pixel in the horizontal direction, the image corresponding to Y is shifted by half a pixel in the vertical direction, and the image corresponding to B is shifted by half a pixel in the horizontal and vertical directions. At this time, the shift pattern during imaging corresponds to FIG. 7. Image data consisting of four images corresponding to RGBY are obtained by these four image pickup elements. In addition, the wavefront modulation element has a phase distribution according to formula (7). The X and Y directions of the wavefront modulation element are approximately 45 degrees to the pixel arrangement direction (x and y directions).

[0059] Here, the Nyquist frequency of each of RGBY corresponds to the broken line 81 in FIG. 9(C). In this case, the direction with the highest Nyquist frequency corresponding to each color of RGBY is the diagonal direction in the lattice of the pixel array of the image sensor. On the other hand, the direction with the highest MTF of the image sensor is determined by the direction around the optical axis of the wavefront modulation element, and is the diagonal direction (diagonal 45 degrees) expressed by Y=X using X and Y in formula (7). In order to make the most of the performance of the image sensor, it is preferable that the direction with the highest MTF of the image sensor and the direction with the highest Nyquist frequency of the image coincide with each other. In this embodiment, pixel-enhancing processing is performed for all colors of RGBY. ​​The Nyquist frequency of each of RGBY corresponds to the solid line 83 in FIG. 9(D). That is, the Nyquist frequency as an image (the Nyquist frequency in each image included in the image data) is the highest in the diagonal direction. In addition, by performing pixel-enhancing processing using images of different colors that are shifted in the diagonal direction, the direction with the highest Nyquist frequency before and after the image processing also coincides. Therefore, in the arrangement of the image sensor and the high pixel count processing of this embodiment, by arranging the pixels so that their array directions (x direction, y direction) coincide with the Y=X direction (45 degree diagonal) of the wavefront modulation element, the performance of the image sensor can be maximized, thereby obtaining a high pixel count image with a higher sense of resolution.

[0060] Compared with the first embodiment, it can be seen that even if the orientation of the imaging element is the same, the direction in which the Nyquist frequency of the highly pixelated image is the largest changes depending on the shift direction when acquiring the captured image used in the high pixel processing. In this embodiment, there are three types of shifts with respect to the reference image, and the orientation of the wavefront modulation element may be set to the horizontal or vertical direction, which is the shortest shift direction. For the smallest shift amount, that is, the shortest shift direction, the diagonal sampling is the finest. Therefore, by setting the direction in which the MTF of the imaging optical system is the largest to approximately 45 degrees with respect to the shortest shift direction, the imaging element can make the most of the performance of the imaging optical system to obtain a highly pixelated image with a higher sense of resolution.

[0061] The direction (first direction) in which the MTF of the imaging optical system is largest is the two directions, the X direction and the Y direction of the wavefront modulation element (the directions indicated by arrows a and b in FIG. 9C), which are at 45 degrees to the shift direction (the second direction (the directions indicated by arrows c and d)).

[0062] In addition, it is preferable that the sampling periods of all images included in the captured image data are equal. If the sampling periods are different, the shift direction in each local region of the image changes, and the direction in which the Nyquist frequency of the high-pixel image is the largest also changes. By making the sampling periods equal, the shift direction can be uniquely determined, and an optimal sense of resolution can be obtained over the entire image by arranging the direction in which the MTF of the imaging optical system is the largest and the direction in which the shift is the shortest among the shifts at approximately 45 degrees.

[0063] It is also preferable that the directions of the highest Nyquist frequencies corresponding to the colors included in the image data are all the same. By making the directions of the highest Nyquist frequencies the same for each color, the directions of the highest Nyquist frequencies of the highly pixelated image are the same for all colors. Therefore, an arrangement in which the direction of the highest MTF of the imaging optical system and the direction of the shortest shift among the shifts are at approximately 45 degrees can be optimal for all colors.

[0064] Also, there may be a mode in which an output image is acquired using an image captured by relatively shifting the acquired image data, and a mode in which the output image is acquired without using the image captured by shifting. In this case, the former performs high pixel processing, but the latter does not. In the imaging by the shift pattern of the first embodiment, the direction in which the MTF of the imaging optical system is the largest is different between the former and the latter. Therefore, it is preferable to determine the direction of the wavefront modulation element depending on the mode. In this case, the wavefront modulation element may be driven, or an instruction may be given to the user to change the direction of the wavefront modulation element.

[0065] The wavefront modulation element and the imaging unit may be detachable. When an imaging unit in which a plurality of imaging elements are shifted and arranged is used, the shift pattern differs depending on the imaging unit. Therefore, by changing the direction of the wavefront modulation element depending on the imaging unit, each imaging unit can be arranged so that the optical performance of the imaging optical system can be utilized. In particular, an image captured by an imaging unit that does not shift the imaging element and a highly pixelated image captured by an imaging unit in which the imaging element is shifted only in the diagonal direction of the pixel array lattice have different directions in which the Nyquist frequency is greatest. Therefore, in an imaging optical system in which both of these can be attached, it is preferable to determine the direction of the wavefront modulation element by the imaging unit. In this case, the wavefront modulation element may be driven, or an instruction may be given to a user to change the direction of the wavefront modulation element.

[0066] In each embodiment, an input image may be output from the imaging device to an image processing device provided separately from the imaging device, and image processing may be performed by the image processing device. Also, the image restoration processing and high pixel processing in each embodiment do not need to be performed by a single image processing device. For example, a program for realizing part or all of the processing in each embodiment may be supplied to one or more systems or devices via a network or a recording medium, and the program may be executed by the system or device.

[0067] As described above, in each embodiment, the imaging device 100 includes an imaging optical system (optical system 101) including a wavefront modulation element 101c that is rotationally asymmetric with respect to the optical axis, and an imaging unit 102 that photoelectrically converts an optical image formed by the imaging optical system. The imaging unit acquires first image data corresponding to a first color in the first imaging, and acquires second image data corresponding to a second color in the second imaging (image data corresponding to multiple colors is acquired by acquiring multiple images corresponding to single colors acquired for each imaging). The second image data is image data that is imaged by shifting the first image data by a non-integer pixel of the imaging unit (at least one image among the image data is imaged by shifting the non-integer pixel with respect to a reference image that is a reference among the image data). In addition, the angle between the first direction in which the modulation transfer function MTF of the imaging optical system is the largest and the second direction, which is the shortest among the shift directions, is 30 degrees to 60 degrees (approximately 45 degrees).

[0068] Preferably, the second direction is different from the pixel array directions (x direction and y direction) of the imaging section. Also preferably, the sampling periods (pixel pitch) of the first image data and the second image data are equal to each other. Also preferably, the directions of the highest Nyquist frequencies in the first image data and the second image data are equal to each other. Also preferably, the first direction and the direction of the highest Nyquist frequency corresponding to at least one image sensor of the imaging section are different from each other. Also preferably, the first direction and the direction of the highest Nyquist frequency corresponding to at least one of the first color or the second color are different from each other. Also preferably, the first color is R or B, and the second color is G.

[0069] Preferably, the imaging device has an image processing unit 104 that generates a high-pixel image using the first image data and the second image data. More preferably, the image processing unit performs an image restoration process using the first image data and the second image data. Also preferably, the image processing unit has a mode for generating an output image. The mode includes a first mode for generating an output image using both the first image data and the second image data, and a second mode for generating an output image using one of the first image data or the second image data. More preferably, the direction of the wavefront modulation element is determined based on the mode.

[0070] Preferably, the wavefront modulating element modulates the wavefront equally along a first axis and a second axis that are orthogonal to each other, and has an odd function shape along each of the first axis and the second axis. Note that the odd function shape is, for example, a shape expressed by Equation (7), but is not limited thereto. Preferably, the imaging optical system and the imaging unit are detachable. In this case, for example, when the imaging optical system is attached to an imaging unit that does not acquire image data including a relatively shifted image, the wavefront modulation element is arranged in a different direction from when the imaging optical system is attached to an imaging unit that acquires image data including a relatively shifted image. Also preferably, the imaging unit has a first imaging element that receives light corresponding to a first color and a second imaging element that receives light corresponding to a second color. In this case, for example, the second imaging element is arranged so as to acquire an image of the subject at a position that differs from the first imaging element by a non-integer pixel of the pixel array. Alternatively, preferably, the imaging unit has one imaging element that receives both light corresponding to the first color and light corresponding to the second color.

[0071] According to each embodiment, it is possible to provide an imaging device capable of obtaining a high-quality image by using a depth expanding optical system.

[0072] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0073] 100 Imaging device 101 Imaging optical system 101c Wavefront Modulation Element 102 Imaging unit

Claims

1. an imaging optical system including a wavefront modulation element that is rotationally asymmetric with respect to an optical axis; an imaging unit that photoelectrically converts an optical image formed by the imaging optical system, The imaging unit acquires first image data corresponding to a first color and one or more shifted image data including second image data corresponding to a second color; the one or more shifted image data are each acquired by imaging in which the optical image is shifted by a shift amount of a non-integer pixel of the imaging unit with respect to the first image data, A direction in which the modulation transfer function of the imaging optical system is the largest is defined as a first direction, a shift direction corresponding to the smallest shift amount among the shift amounts of each of the one or more shift image data is defined as a second direction, and an angle between the first direction and the second direction is defined as θ [°]. 30≦θ≦60 An imaging apparatus characterized in that the following condition is satisfied.

2. The imaging device according to claim 1 , wherein the second direction is different from a pixel array direction of the imaging section.

3. 3. The imaging apparatus according to claim 1, wherein the first image data and the second image data are sampled at the same period.

4. 4. The imaging device according to claim 1, wherein the first image data and the second image data have the same direction in which the Nyquist frequency is highest.

5. 5. The imaging device according to claim 1, wherein the first direction and a direction in which a Nyquist frequency corresponding to at least one image sensor of the imaging section is highest are different from each other.

6. 6. The imaging device according to claim 1, wherein the first direction and a direction in which a Nyquist frequency corresponding to at least one of the first color and the second color is highest are different from each other.

7. the first color is R or B; 7. The imaging device according to claim 1, wherein the second color is G.

8. 8. The imaging device according to claim 1, further comprising an image processing unit that generates a high-pixel image by using the first image data and the second image data.

9. 9. The imaging apparatus according to claim 8, wherein the image processing unit performs an image restoration process using the first image data and the second image data.

10. the image processor has a mode for generating an output image; The mode is a first mode for generating the output image using the first image data and the second image data; 10. The imaging apparatus according to claim 8, further comprising a second mode in which the output image is generated using one of the first image data and the second image data.

11. 11. The imaging apparatus according to claim 10, wherein a rotation direction of the wavefront modulation element with respect to the optical axis is determined based on the mode.

12. 12. The imaging device according to claim 1, wherein the wavefront modulation element modulates the wavefront equally along a first axis and a second axis that are mutually orthogonal, and has an odd function shape along each of the first axis and the second axis.

13. 13. The imaging device according to claim 1, wherein the imaging optical system and the imaging unit are detachable.

14. The imaging unit includes: a first image sensor that receives light corresponding to the first color; 14. The imaging device according to claim 1, further comprising a second imaging element that receives light corresponding to the second color.

15. 14. The imaging device according to claim 1, wherein the imaging section includes one imaging element that receives light corresponding to the first color and light corresponding to the second color.

Images