IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

Abstract

To provide an image processing device capable of mitigating blur due to an aberration and diffraction of an optical system while decreasing a volume of data and arithmetic operation when performing a high pixelation process and an image restoration process.SOLUTION: An image processing device (320) includes: an image acquisition unit (322) which acquires a first image and a second image respectively having different imaging positions from each other; an image restoration unit (324) which performs image restoration process to the first image and the second image, and acquires each of a first restored image and a second restored image; and a high pixelation process unit (325) which acquires a high pixelation image by using the first restored image and the second restored image.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image processing device that performs image high-resolution processing and image restoration processing. [Background technology]

[0002] Conventionally, pixel-enhancing processing that uses pixel-shifted images to increase the number of pixels of an image beyond the number of pixels of an imaging element is known. Patent Document 1 discloses an image processing method that synthesizes a plurality of image data acquired by performing pixel shifting to generate a high-resolution composite image (high-pixel image). Patent Document 1 also discloses an image processing method that performs edge enhancement on the composite image based on the MTF characteristics of the imaging optical system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-13046 A Summary of the Invention [Problem to be solved by the invention]

[0004] When performing image restoration processing on a highly pixelated image by the image processing method disclosed in Patent Document 1, the number of taps required for the image restoration filter required for the image restoration processing becomes large. For example, when applying an image restoration filter to a highly pixelated image with four times the number of pixels, in order to correct blurring in the same range as that of the image restoration filter for an image obtained by capturing, four times the number of taps is required. Therefore, when performing image restoration processing on a highly pixelated image, the amount of data and the amount of calculation increase.

[0005] Therefore, an object of the present invention is to provide an image processing device, an imaging device, an image processing method, and a program that can reduce the amount of data and calculations while reducing blurring caused by aberrations and diffraction in the optical system when performing high-pixel processing and image restoration processing. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an image processing apparatus, Multiple Images An image acquisition unit for acquiring the image Multiple Images By performing recovery processing on Multiple recovery images A first processing unit for acquiring the Multiple recovery images and a second processing unit that acquires a third image using the first processing unit performs a recovery process on the plurality of images after all of the plurality of images are acquired by the image acquisition unit; The number of pixels of the third image is Multiple Images More than each of them.

[0007] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, and a program that can reduce the amount of data and the amount of calculations while reducing blurring due to aberrations and diffraction in the optical system when performing high-pixel processing and image restoration processing. [Brief description of the drawings]

[0009] [Figure 1] 5 is an explanatory diagram of an image restoration filter in each embodiment. FIG. [Diagram 2] 4A to 4C are explanatory diagrams (sectional views) of an image restoration filter in each embodiment. [Diagram 3] FIG. 4 is an explanatory diagram of a point spread function PSF in each embodiment. [Figure 4] 4A to 4C are explanatory diagrams of an amplitude component MTF and a phase component PTF of an optical transfer function in each embodiment. [Diagram 5] 5A to 5C are explanatory diagrams of pixel-enhancing processing in each embodiment. [Figure 6] 11A to 11C are explanatory diagrams of other pixel-enhancing processes in the respective embodiments. [Figure 7] 4 is a flowchart of an image processing method in the first embodiment. [Figure 8]4 is an explanatory diagram of a method for generating an image restoration filter in the first embodiment. FIG. [Figure 9] 4 is an explanatory diagram of a restoration gain of an image restoration filter in the first embodiment. FIG. [Figure 10] 5 is an explanatory diagram of a target MTF when generating an image restoration filter in the first embodiment. FIG. [Figure 11] 10 is a flowchart of an image processing method according to a second embodiment. [Figure 12] FIG. 11 is an explanatory diagram of an image processing system according to a third embodiment. [Figure 13] FIG. 13 is a block diagram of an imaging device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] A captured image obtained by an imaging device contains blur components and is degraded due to the influence of various aberrations of the imaging optical system, such as spherical aberration, coma aberration, field curvature, astigmatism, etc. Such image blur components caused by aberrations mean that a light beam emitted from a single point on the subject should converge again to a single point on the imaging surface in the absence of aberration and the influence of diffraction, but instead spreads out, and are expressed by a point spread function (PSF).

[0012] The optical transfer function OTF (Optical Transfer Function) obtained by Fourier transforming the point spread function PSF is frequency component information of aberration and is expressed as a complex number. The absolute value of the optical transfer function OTF, i.e., the amplitude component, is called the modulation transfer function (MTF), and the phase component is called the phase transfer function (PTF). The amplitude component MTF and phase component PTF are the frequency characteristics of the amplitude component and phase component of image degradation caused by aberration, respectively, and are expressed by the following formula with the phase component as the phase angle.

[0013] PTF=tan -1(Im(OTF) / Re(OTF)) Here, Re(OTF) and Im(OTF) respectively represent the real part and the imaginary part of the optical transfer function OTF. In this way, the optical transfer function OTF of the imaging optical system deteriorates the amplitude component MTF and phase component PTF of the image, so that each point of the subject in the deteriorated image is asymmetrically blurred like coma aberration. In addition, magnification chromatic aberration occurs when the imaging position shifts due to the difference in imaging magnification for each wavelength of light, and this is acquired as, for example, RGB color components according to the spectral characteristics of the imaging device. Therefore, the imaging position shifts between RGB, and within each color component, the imaging position shifts for each wavelength, i.e., the image spreads due to phase shift.

[0014] As a method for correcting the deterioration of the amplitude component MTF and phase component PTF, a method using information on the optical transfer function OTF of the imaging optical system is known. This method is called image recovery or image restoration, and hereinafter, the process of correcting the deterioration of a captured image using information on the optical transfer function (OTF) of the imaging optical system is called image restoration processing. The details will be described later, but one of the image restoration methods known is a convolution method in which an input image is convolved with an image restoration filter having the inverse characteristics of the optical transfer function (OTF).

[0015] To effectively use image restoration, it is necessary to obtain more accurate OTF information of the imaging optical system. The OTF of a general imaging optical system varies greatly depending on the image height (position of the image). In addition, the optical transfer function OTF is two-dimensional data and a complex number, so it has a real part and an imaginary part. In addition, when performing image restoration processing on a color image having three color components of RGB, the OTF data for one image height is the number of taps in the vertical direction × the number of taps in the horizontal direction × 2 (real part, imaginary part) × 3 (RGB). Here, the number of taps is the vertical and horizontal size of the OTF data. If these are stored for all shooting conditions such as image height, F-number (aperture value), zoom (focal length), and shooting distance, the amount of data will be enormous.

[0016] On the other hand, there is a pixel-enhancing process for increasing the number of pixels of an image to be greater than the number of pixels of an image sensor. As a pixel-enhancing process, a pixel-enhancing method (pixel-enhancing process) using a pixel-shifted image is known.

[0017] First, the definitions of terms used in each embodiment, and the high pixel processing and image restoration processing (image processing method) will be described. The image processing method described here will be used as appropriate in each embodiment described later.

[0018] [Photo] A captured image is a digital image obtained by receiving light with an image sensor through an imaging optical system, and is degraded by the optical transfer function (OTF) caused by aberrations in the imaging optical system, which includes lenses and various optical filters. The imaging optical system can be configured using not only lenses but also mirrors (reflective surfaces) with curvature.

[0019] The color components of the captured image include, for example, information on RGB color components. As color components, other commonly used color spaces such as lightness, hue, and saturation expressed by LCH, and luminance and color difference signals expressed by YCbCr can be selected and used. Other color spaces that can be used include XYZ, Lab, Yuv, and JCh. Furthermore, color temperature may be used.

[0020] The captured image (input image) and the output image can be accompanied by shooting conditions such as the focal length of the lens, the aperture value, and the shooting distance, as well as various correction information for correcting the image. When an image is transferred from the imaging device to another image processing device for correction processing, it is preferable to attach information about the shooting conditions and correction to the captured image as described above. As another method of transferring information about the shooting conditions and correction, the imaging device and the image processing device may be directly or indirectly connected to each other and the information may be transferred.

[0021] [Image recovery processing] Next, an overview of the image restoration process will be explained. When the captured image (degraded image) is g(x,y), the original image is f(x,y), and the point spread function PSF, which is a Fourier pair of the optical transfer function OTF, is h(x,y), the following formula (1) is established.

[0022] g(x,y)=h(x,y)*f(x,y) … (1) Here, * denotes convolution (convolution integral, product sum), and (x, y) are coordinates on the captured image.

[0023] Furthermore, if equation (1) is Fourier transformed and converted into a frequency domain display format, equation (2), expressed as a product for each frequency, is obtained.

[0024] G(u,v)=H(u,v)·F(u,v) … (2) Here, H is the optical transfer function OTF obtained by Fourier transforming the point spread function PSF(h), and G and F are functions obtained by Fourier transforming the degraded image g and the original image f, respectively. (u,v) are the coordinates in the two-dimensional frequency plane, i.e., the frequency.

[0025] To obtain the original image f from the captured degraded image g, it is sufficient to divide both sides of the following equation (3) by the optical transfer function H.

[0026] G(u,v) / H(u,v)=F(u,v) … (3) Then, F(u,v), i.e., G(u,v) / H(u,v), is transformed back to the real plane by inverse Fourier transform, and the original image f(x,y) is obtained as a restored image.

[0027] If the inverse Fourier transform of H-1 is R, the original image f(x, y) can be obtained by performing convolution processing on the image on the real surface as shown in the following equation (4).

[0028] g(x,y)*R(x,y)=f(x,y) … (4) Here, R(x,y) is called an image restoration filter. When the image is a two-dimensional image, the image restoration filter R is generally a two-dimensional filter having taps (cells) corresponding to each pixel of the image. In addition, the more taps (cells) the image restoration filter R has, the higher the restoration accuracy is. For this reason, a feasible number of taps is set according to the required image quality, image processing capability, aberration characteristics, and the like. The image restoration filter R must at least reflect the characteristics of aberration, and therefore is different from conventional edge enhancement filters with about three taps each for the horizontal and vertical directions. The image restoration filter R is set based on the optical transfer function OTF, and therefore can correct both the deterioration of the amplitude component and the phase component with high accuracy.

[0029] Furthermore, since actual images contain noise components, if the image restoration filter R created by taking the reciprocal of the optical transfer function OTF as described above is used, the noise components will be significantly amplified while restoring the degraded image. This is because the MTF (amplitude component) of the optical system is raised to return it to 1 across all frequencies in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is the amplitude degradation caused by the optical system, returns to 1, but at the same time the power spectrum of the noise is also raised, and as a result, the noise is amplified according to the degree to which the MTF is raised (restoration gain).

[0030] Therefore, when noise is included, an image of good quality for viewing cannot be obtained. This is expressed by the following equations (5-1) and (5-2).

[0031] G(u,v)=H(u,v)·F(u,v)+N(u,v) … (5-1) G(u,v) / H(u,v)=F(u,v)+N(u,v) / H(u,v) … (5-2) Here, N is the noise component.

[0032] For images containing noise components, there is a method for controlling the degree of restoration according to the intensity ratio SNR of an image signal to a noise signal, such as a Wiener filter expressed by the following equation (6).

[0033]

number

[0034] Here, M(u,v) is the frequency characteristic of the Wiener filter, and |H(u,v)| is the absolute value of the optical transfer function OTF (MTF). In this method, for each frequency, the smaller the MTF is, the smaller the restoration gain (degree of restoration) is, and the larger the MTF is, the larger the restoration gain is. Generally, the MTF of an imaging optical system is high on the low frequency side and low on the high frequency side, so this method essentially reduces the restoration gain on the high frequency side of the image.

[0035] Next, the image restoration filter will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is an explanatory diagram of the image restoration filter. The number of taps of the image restoration filter is determined according to the aberration characteristics of the imaging optical system and the required restoration accuracy. The image restoration filter of Fig. 1 is, as an example, a two-dimensional filter with 11 × 11 taps. Although the values ​​(coefficients) in each tap are omitted in Fig. 1, a cross section of this image restoration filter is shown in Fig. 2. The distribution of the values ​​(coefficient values) of each tap of the image restoration filter has the function of ideally returning the signal value (PSF) that has spread spatially due to aberration to the original single point.

[0036] Each tap of the image restoration filter is subjected to convolution processing (convolution integral, product-sum) in the image restoration process corresponding to each pixel of the image. In the convolution processing, in order to improve the signal value of a specific pixel, the pixel is aligned with the center of the image restoration filter. Then, for each corresponding pixel of the image and the image restoration filter, the product of the image signal value and the filter coefficient value is calculated, and the sum of these products is replaced as the signal value of the center pixel.

[0037] Next, the characteristics of image restoration in real space and frequency space will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is an explanatory diagram of the point spread function PSF, where Fig. 3(a) shows the point spread function PSF before image restoration, and Fig. 3(b) shows the point spread function PSF after image restoration. Fig. 4 is an explanatory diagram of the amplitude component MTF (Fig. 4(M)) and phase component PTF (Fig. 4(P)) of the optical transfer function OTF. The dashed line (a) in Fig. 4(M) shows the MTF before image restoration, and the dashed line (b) shows the MTF after image restoration. The dashed line (a) in Fig. 4(P) shows the PTF before image restoration, and the dashed line (b) shows the PTF after image restoration. As shown in Fig. 3(a), the point spread function PSF before image restoration has an asymmetric spread, and due to this asymmetry, the phase component PTF has a nonlinear value with respect to frequency. In the image restoration process, the amplitude component MTF is amplified and the phase component PTF is corrected to zero, so that the point spread function PSF after image restoration has a symmetric and sharp shape.

[0038] In this way, the image restoration filter can be obtained by performing an inverse Fourier transform on a function designed based on the inverse function of the optical transfer function OTF of the imaging optical system. The image restoration filter used in this embodiment can be changed as appropriate, and for example, the Wiener filter described above can be used. When using a Wiener filter, it is possible to create a real-space image restoration filter that is actually convolved with the image by performing an inverse Fourier transform on Equation (6).

[0039] In addition, since the optical transfer function (OTF) due to aberration changes depending on the image height (image position) of the imaging optical system even in one shooting state, the image restoration filter needs to be changed depending on the image height.

[0040] On the other hand, the optical transfer function (OTF) due to diffraction, whose influence becomes dominant as the F-number increases, can be treated as a uniform OTF for all image heights if the influence of vignetting in the optical system is small.

[0041] When the correction target of the image restoration process is diffraction (diffraction blur) without including aberration, the image restoration filter depends only on the aperture value and the wavelength of light, and does not depend on the image height (position of the image). Therefore, a uniform (fixed) image restoration filter can be used for one image. In other words, an image restoration filter for correcting diffraction blur is generated based on an optical transfer function due to the diffraction blur generated according to the aperture value. Regarding wavelength, optical transfer functions at multiple wavelengths are calculated, and an optical transfer function for each color component can be generated by weighting for each wavelength based on the spectrum of the assumed light source and the light receiving sensitivity information of the image sensor. Alternatively, calculations may be performed at a representative wavelength for each predetermined color component. Then, an image restoration filter can be generated based on the optical transfer function for each color component.

[0042] Therefore, when only diffraction is to be corrected, a plurality of image restoration filters that depend on the aperture value are stored in advance, and the image can be processed using a uniform (fixed) image restoration filter according to the shooting condition of the aperture value. It is also preferable to take into account the aperture degradation component caused by the shape of the pixel aperture and the characteristics of the optical low pass filter.

[0043] The above-mentioned image restoration process is also performed when a phase mask (wavefront modulation element) that reduces the performance variation in the depth direction is inserted into the optical system and a depth-enlarging method (WFC: WaveFront Coding) is performed. When a phase mask is inserted into the optical system to reduce the performance variation in the depth direction, the performance variation also tends to reduce with respect to the imaging conditions such as image height and object distance. In other words, when a phase mask is inserted into the optical system and there is little performance variation with respect to the image height, it can be treated as a uniform OTF with respect to the image height, as in the case of correcting diffraction blur. Therefore, even when the optical system includes a phase mask and depth-enlarging is performed, a uniform (fixed) image restoration filter may be used for one image. For example, a cubic phase mask whose surface shape is expressed by a cubic function is used as the phase mask. The phase mask used for depth-enlarging is not limited to the above, and various methods may be used.

[0044] Next, an overview of the pixel-enhancing process will be described with reference to FIG. 5. FIG. 5 is an explanatory diagram of the pixel-enhancing process, and shows an example of a captured image used in the pixel-enhancing process. In the pixel-enhancing process, a pixel-enhancing image is obtained from a plurality of captured images taken at different shooting positions. In FIG. 5, four captured images used in the pixel-enhancing process are obtained by taking images with the shooting positions shifted by half a pixel from each other. Each of the four captured images is obtained with a reference state 51 (no shift), a horizontal shift 52, a vertical shift 53, and a horizontal and vertical shift 54. By obtaining the four captured images shifted by half a pixel, a pixel-enhancing image 55 with four times the number of pixels can be obtained by the pixel-enhancing process. The shifting of the shooting position may be performed by shifting the image sensor, or may be performed by moving the optical system, some lenses included in the optical system, or the entire imaging device.

[0045] 5 shows an example in which the image sensor is a monochrome sensor, but if the image sensor has a Bayer array, multiple captured images shifted by one pixel may be acquired, and then multiple captured images shifted by a half pixel may be acquired. By capturing images shifted by one pixel when the image sensor has a Bayer array, the luminance value of each color can be accurately acquired at each pixel without performing demosaic processing by interpolation.

[0046] 6 is an explanatory diagram of another high pixel processing, showing an example in which a plurality of image pickup elements are used to obtain a plurality of images to be used in the high pixel processing. For example, when capturing images using different image pickup elements for RGB, the shooting position of G is shifted by half a pixel from the shooting positions of R and B to obtain a captured image to be used in the high pixel processing. Then, the high pixel processing is performed by an interpolation process, thereby obtaining a high pixel image 61 having a greater number of pixels than the captured image. Note that the high pixel processing and the capturing of the images to be used therein are not limited to those described above, and other methods may be used. EXAMPLES

[0047] Next, an image processing method according to the first embodiment of the present invention will be described with reference to Fig. 7. Fig. 7 is a flowchart of the image processing method (image processing program) according to this embodiment. The image processing method according to this embodiment is executed by a computer constituted by a CPU and the like as an image processing device, according to an image processing program as a computer program. This is the same for the other embodiments described later.

[0048] First, in step S11, the image processing device acquires a captured image generated by the imaging device through imaging. The captured image may be acquired from the imaging device through wired or wireless communication between the imaging device and the image processing device, or through a storage medium such as a semiconductor memory or an optical disk. Here, when acquiring the captured image, images are acquired with the image capturing positions shifted from each other in order to perform the high pixel processing described later. As described above, the image to be used for the high pixel processing can be captured using various methods. In this embodiment, as an example, four images captured with the image capturing positions shifted from each other by half a pixel (reference state, shifted in the horizontal direction, shifted in the vertical direction, shifted in the horizontal and vertical directions) as shown in FIG. 5 are acquired as the captured images.

[0049] Next, in step S12, the image processing device acquires an image restoration filter to be used in the image restoration process described later. In this embodiment, an example is described in which aberration information (optical information) is acquired based on the photographing condition, and an image restoration filter is acquired based on the aberration information. That is, in this embodiment, the image restoration filter is generated based on the optical information of the imaging optical system.

[0050] First, the image processing device acquires the shooting conditions (shooting condition information) when the imaging device generates a captured image by capturing an image. As described above, the shooting conditions include the focal length, aperture value (F value), and shooting distance of the imaging optical system, as well as identification information (camera ID) of the imaging device. In addition, in an imaging device in which the imaging optical system can be replaced, the shooting conditions may include identification information (lens ID) of the imaging optical system (interchangeable lens). The shooting condition information may be acquired as information attached to the captured image as described above, or may be acquired via wired or wireless communication or a storage medium.

[0051] Next, the image processing device acquires aberration information suitable for the shooting conditions. In this embodiment, the aberration information is an optical transfer function OTF. The image processing device selects and acquires an appropriate optical transfer function OTF according to the shooting conditions from a plurality of optical transfer functions OTF stored in advance. In addition, when the shooting conditions such as the aperture value, shooting distance, and focal length of the zoom lens are specific shooting conditions, the optical transfer function OTF corresponding to the shooting condition can also be generated by an interpolation process from the optical transfer functions OTF of other shooting conditions stored in advance. In this case, it is possible to reduce the amount of data of the optical transfer function OTF stored. For example, bilinear interpolation (linear interpolation) and bicubic interpolation are used as the interpolation process, but the present invention is not limited thereto.

[0052] In this embodiment, the image processing device acquires an optical transfer function OTF as the aberration information, but is not limited to this. Instead of the optical transfer function OTF, aberration information such as a point spread function PSF may be acquired. In addition, in this embodiment, the image processing device may acquire coefficient data that approximates the aberration information by fitting to a predetermined function, and reconstruct the optical transfer function OTF and the point spread function PSF from the coefficient data. For example, the optical transfer function OTF may be fitted using the Legendre polynomial. Also, fitting may be performed using other functions such as the Chebushev polynomial.

[0053] In this embodiment, the image processing device generates a plurality of optical transfer functions OTF in one direction passing through the center of the screen (the center of the captured image) or the optical axis of the imaging optical system. The imaging optical system may include an imaging element, an optical low-pass filter, etc.

[0054] Next, the image processing device rotates the optical transfer function OTF around the screen center (center of the captured image) or the optical axis of the imaging optical system to expand the optical transfer function OTF. Specifically, the image processing device interpolates the optical transfer function OTF in accordance with the pixel array, thereby discretely disposing the optical transfer function OTF at multiple positions in the captured image.

[0055] Next, the image processing device converts the optical transfer function OTF into an image restoration filter, i.e., generates an image restoration filter using the expanded optical transfer function OTF. The image restoration filter is generated by creating a restoration filter characteristic in frequency space based on the optical transfer function OTF and converting it into a filter in real space (image restoration filter) by inverse Fourier transform.

[0056] 8(a) to (e) are explanatory diagrams of a method for generating an image restoration filter. As shown in Fig. 8(a), the optical transfer function OTF is arranged in the area of ​​the circumscribed circle of the image (imaging area) in one direction (vertical direction) passing through the center of the screen (center of the captured image) or the optical axis of the imaging optical system.

[0057] In this embodiment, the optical transfer function is expanded on a straight line as shown in FIG. 8(a), but is not limited thereto. For example, in the captured image plane, straight lines passing through the center of the captured image or the optical axis of the imaging optical system and perpendicular to each other are defined as a first straight line (y in FIG. 8(a)) and a second straight line (x in FIG. 8(a)). In this case, at least two of the acquired optical transfer functions may be optical transfer functions corresponding to the position (image height) on the first straight line. That is, the optical transfer functions OTF may not be linearly arranged in one direction as long as they are arranged at multiple positions (multiple positions in the captured image) arranged at different distances from the center of the screen or the optical axis of the imaging optical system in a predetermined direction. Note that when there is no pixel including the center of the captured image or the optical axis of the imaging optical system, that is, when the center of the captured image or the optical axis of the imaging optical system is between pixels, the acquired optical transfer function may be an optical transfer function corresponding to the position (image height) of the pixel sandwiching the first straight line.

[0058] In addition, when the optical transfer functions OTF are arranged in one direction, they are not limited to the vertical direction and may be arranged in other directions such as the horizontal direction. It is more preferable to arrange the optical transfer functions OTF linearly in either the vertical direction or the horizontal direction, since this embodiment of the image processing can be performed more easily.

[0059] Next, the optical transfer function OTF is rotated, and if necessary, an interpolation process (various processes according to the pixel arrangement after rotation) is performed, and the optical transfer function OTF is rearranged as shown in Fig. 8(b). The interpolation process includes an interpolation process in the radial direction and an interpolation process associated with the rotation, and the optical transfer function OTF can be rearranged to an arbitrary position. Next, for the optical transfer function OTF at each position, the frequency characteristics of the image restoration filter are calculated, for example, as shown in Equation (6), and an inverse Fourier transform is performed, thereby converting it into an image restoration filter in real space as shown in Fig. 8(c).

[0060] That is, in the photographed image, the straight lines passing through the center of the photographed image or the optical axis of the imaging optical system and perpendicular to each other are defined as the first straight line (y in FIG. 8(a)) and the second straight line (x in FIG. 8(a)). The area that is point-symmetrical with the first area (83 in FIG. 8(c)) of the photographed image with respect to the center of the photographed image or the optical axis OA of the imaging optical system is defined as the second area (81 in FIG. 8(c)). In addition, the area that is line-symmetrical with respect to the first area and the first straight line is defined as the third area (82 in FIG. 8(c)), and the area that is line-symmetrical with respect to the first area and the second straight line is defined as the fourth area (84 in FIG. 8(c)). At this time, the optical transfer function of the first area is used to generate the optical transfer function of the second area, the third area, and the fourth area. This reduces the Fourier transform processing to approximately 1 / 4 of the position to be finally rearranged. Moreover, if the optical transfer function OTF in Fig. 8(b) and the image restoration filter in Fig. 8(c) are rearranged by rotation and interpolation processing as shown in Fig. 8(e) and expanded as shown in Fig. 8(d) using the symmetry, the Fourier transform processing can be further reduced. Note that the arrangement (arrangement density of the restoration filters) shown in Fig. 8(a) to (e) is one example, and the arrangement interval can be set arbitrarily according to the fluctuation of the optical transfer function OTF of the imaging optical system.

[0061] In this embodiment, the optical transfer function OTF arranged in one direction passing through the center of the screen or the optical axis of the imaging optical system as described above is rotated and expanded on the premise that the optical transfer function OTF is rotationally symmetric with respect to the center of the imaging surface (center of the screen) or the optical axis of the imaging optical system. This makes it possible to perform image restoration processing with a small amount of data. In addition, when the correction target of the image restoration processing is blur that does not include aberration and does not depend on the image height (position of the image), such as diffraction (diffraction blur), a uniform (fixed) optical transfer function OTF or image restoration filter may be used for one image.

[0062] The image restoration filter acquired in step S12 above is applied to the captured image in step S13 described below. Therefore, it is difficult to perform ideal image restoration processing for frequencies higher than the Nyquist frequency generated by the high pixel processing described below. Signals with frequencies higher than the Nyquist frequency are folded back to within the Nyquist frequency at the stage of performing image restoration processing on the captured image, and a restoration gain at the folded back frequency is applied.

[0063] FIG. 9 is an explanatory diagram of the restoration gain and the ideal restoration gain in the image restoration process for the captured image. In FIG. 9, the horizontal axis indicates frequency, and the vertical axis indicates restoration gain. The frequency on the horizontal axis indicates frequencies up to twice the Nyquist frequency, half the frequency up to the Nyquist frequency, and the frequency band thereafter generated by the high pixel processing. In contrast to the ideal gain 91 in a state including frequencies after the Nyquist frequency generated by the high pixel processing, the restoration gain up to the Nyquist frequency is applied in the image restoration process for the captured image. In other words, the restoration gain 92 at the frequency folded back to within the Nyquist frequency is applied to the signal after the Nyquist frequency. Therefore, it is difficult to apply the ideal restoration gain to the frequency after the Nyquist frequency generated by the high pixel processing.

[0064] Therefore, the restoration gain of the image restoration filter is set based on the restoration characteristics at frequencies equal to or lower than the Nyquist frequency and the restoration characteristics at frequencies higher than the Nyquist frequency, i.e., the image restoration filter is generated based on the restoration characteristics at frequencies higher than the Nyquist frequency in addition to the restoration characteristics at frequencies equal to or lower than the Nyquist frequency.

[0065] FIG. 9 shows an example of setting the restoration gain. For example, the restoration gain 94 is set so that the average or weighted average restoration gain 93 of the ideal restoration gain 91 after the Nyquist frequency and the restoration gain 92 below the Nyquist frequency applied to the frequency after the Nyquist frequency is applied to the frequency after the Nyquist frequency. Specifically, the ratio between the restoration gain 92 below the Nyquist frequency applied after the Nyquist frequency and the above-mentioned average restoration gain 93 is calculated, and the value obtained by folding back the ratio is applied to the restoration gain within the Nyquist frequency (indicated by the arrow) to set the restoration gain 94. That is, the image restoration filter is generated based on the optical transfer function (folded optical transfer function) at frequencies higher than the Nyquist frequency in addition to the optical transfer function at frequencies below the Nyquist frequency. This allows signals above the Nyquist frequency generated by the high pixelation process to approach the ideal image restoration process.

[0066] Also, a target MTF for the image restoration process may be set, and an image restoration filter may be generated based on the target MTF. For example, a diffraction limit MTF for a predetermined aperture value may be set as the target MTF. If the target MTF is a high value even after the Nyquist frequency, the target MTF varies greatly depending on the assumed subject. When an object such as a periodic structure of a specific frequency is assumed, the target MTF can be considered simply because there is no aliasing effect of sampling, but for an object including high-frequency components higher than the Nyquist frequency, the target MTF must also be considered with aliasing effect. In other words, the target MTF changes depending on what is assumed as the subject. When an image restoration filter is generated to have a target MTF assuming an object without aliasing effect, appropriate image restoration process is possible for an object such as a periodic structure of a specific frequency, but for an object including high frequencies, the target MTF is not reached and correction is insufficient. When an image restoration filter is generated to have a target MTF assuming an object including high frequencies, over-correction is performed beyond the target MTF for an object such as a periodic structure of a specific frequency. In this way, if there is a discrepancy between the subject assumed when setting the target MTF and the subject to be processed in the image restoration process, under-correction or over-correction may occur.

[0067] Therefore, a target MTF is determined based on the OTF at frequencies below the Nyquist frequency and the OTF at frequencies higher than the Nyquist frequency, and an image restoration filter is generated. That is, a target MTF is determined based on a target MTF below the Nyquist frequency without aliasing and a target MTF when the OTF at frequencies higher than the Nyquist frequency is aliased, and an image restoration filter is generated. FIG. 10 is an explanatory diagram of a target MTF when generating an image restoration filter. For example, as shown in FIG. 10, an image restoration filter is generated with the average or weighted average MTF 103 of a target MTF 101 below the Nyquist frequency without aliasing and a target MTF 102 when the OTF at frequencies higher than the Nyquist frequency is aliased as the target MTF. That is, an image restoration filter is generated that restores the MTF 100 deteriorated by aberrations and the like to the target MTF 103 (indicated by an arrow). This makes it possible to alleviate the above-mentioned undercorrection and overcorrection when a target MTF is set.

[0068] Although the generation and acquisition of the image restoration filter have been described above, the present invention is not limited to this. For example, an image restoration filter may be generated and stored in advance, and the image restoration filter may be acquired based on the shooting conditions.

[0069] 7, the image processing device performs image restoration processing on the captured image acquired in step S11. The image restoration processing is performed based on the image restoration filter acquired in step S12. Here, since the captured images are multiple captured images captured at different shooting positions, the image center is shifted from the optical axis (correction center). Therefore, the image restoration processing may be performed by shifting the correction center according to the shooting position of the captured image to be subjected to the image restoration processing.

[0070] In addition, during the convolution of the image restoration filter, pixels other than the position where the image restoration filter shown in FIG. 8(d) is arranged can be generated by interpolation using multiple filters arranged nearby. In this case, the image restoration filter has a first image restoration filter at a first position of the captured image and a second image restoration filter at a second position of the captured image. The first image restoration filter is generated by using the expanded optical transfer function. The second image restoration filter is generated by interpolating using the first image restoration filter. By performing such an interpolation process, the image restoration filter can be changed for each pixel, for example.

[0071] Next, in step S14, the image processing device acquires a plurality of restored images by performing image restoration processing on a plurality of captured images obtained by capturing images with the image capturing position shifted. Here, the plurality of restored images are acquired by performing image restoration processing based on an image restoration filter on a plurality of captured images obtained by capturing images with the image capturing position shifted in step S11. A plurality of captured images may be acquired and then subjected to image restoration processing to acquire a plurality of restored images, or a plurality of restored images may be acquired by repeatedly performing image restoration processing on a captured image obtained by capturing images with the image capturing position shifted, and then acquiring a restored image.

[0072] Next, in step S15, the image processing device performs pixel-enhancing processing using the multiple restored images acquired in step S14. The pixel-enhancing processing can be performed using various techniques as described above.

[0073] Next, in step S16, the image processing device outputs the pixel-enhanced image obtained by the pixel-enhanced processing as an output image. At this time, the image processing device may perform various processes related to development processing on the pixel-enhanced image.

[0074] The high-pixel image obtained by the above flow is an image in which blurring due to aberration and diffraction is reduced by image restoration processing, and the number of pixels is greater than the number of pixels of the imaging element. In this embodiment, the case where processing based on the inverse function of the optical transfer function OTF is performed as the image restoration processing for correcting image deterioration is described, but the same can be applied to unsharp mask processing using aberration information. In unsharp mask processing, a difference between an unsharp image blurred by applying an unsharp mask to an original image and the original image is added to or subtracted from the original image to generate a sharpened image. At that time, by using the point spread function PSF of the imaging optical system as the unsharp mask, an image in which deterioration due to aberration of the imaging optical system during imaging is corrected can be obtained.

[0075] If the captured image is g(x, y) and the correction component is u(x, y), the corrected image f(x, y) can be expressed by the following equation (7).

[0076] f(x,y)=g(x,y)+m×u(x,y) … (7) In formula (7), the degree of restoration (restoration gain) of the correction component u(x,y) for the captured image g(x,y), that is, the amount of correction, can be adjusted by changing the value of m. Note that the value of m may be changed according to the image height (image position) of the imaging optical system, or may be a constant value.

[0077] Moreover, the correction component u(x, y) is expressed as the following equation (8).

[0078] u(x,y)=g(x,y)-g(x,y)*PSF(x,y) …(8) Moreover, the correction component u(x, y) can be expressed as the following equation (9) by modifying the right side of equation (8).

[0079] u(x,y)=g(x,y)*(δ(x,y)-PSF(x,y)) … (9) In equation (9), δ is a delta function (ideal point spread function). The delta function used here has the same number of taps as PSF(x, y), with the value of the center tap being 1 and the values ​​of all other taps being 0.

[0080] From equations (7) to (9), the corrected image f(x, y) can be expressed as in the following equation (10).

[0081] f(x,y)=g(x,y)*[δ(x,y)+m×(δ(x,y)-PSF(x,y))] … (10) That is, by convolving the part in the curly brackets [ ] in formula (10) as a filter (image restoration filter) with the captured image g(x, y), unsharp mask processing is possible.

[0082] The point spread function PSF changes according to the image height (image position) of the imaging optical system even in one shooting state. Therefore, the filter used for unsharp mask processing is also changed according to the image height. In addition, various high-resolution processing such as super-resolution processing using aberration information can be similarly applied. EXAMPLES

[0083] Next, an image processing method in embodiment 2 of the present invention will be described with reference to Fig. 11. Fig. 11 is a flowchart of the image processing method in this embodiment. In embodiment 1, image restoration processing is performed on the captured image, and then pixel enhancement processing is performed using the restored image. However, this embodiment includes a flow in which pixel enhancement processing is performed using the captured image according to various conditions, and then image restoration processing is performed. The image processing method in this embodiment is executed by a computer according to an image processing program in accordance with the flowchart in Fig. 11. Note that in Fig. 11, steps S11 to S16 are the same as those in embodiment 1 (Fig. 7), and therefore their description will be omitted.

[0084] In step S21, the image processing device determines whether or not to perform the first processing. If the image processing device performs the first processing, it performs the image restoration processing and the pixel enhancement processing based on the flowchart of the first embodiment (FIG. 7). On the other hand, if the image processing device does not perform the first processing, it proceeds to step S22. In the flow after step S22, after performing the pixel enhancement processing using the captured image, it performs the image restoration processing (second processing).

[0085] The process of determining whether to perform the first process is performed, for example, based on the recovery target data. When the recovery target data is a moving image, the first process is performed, and when the recovery target data is a still image or a still image cut out from a moving image, the second process is performed. When the recovery target is a moving image, an image recovery filter with a small number of taps is used to reduce the amount of data and the amount of calculation, and processing speed is prioritized, while for still images, image recovery processing that takes into account high frequencies generated by high pixel processing, although it requires high processing speed, can be performed. Alternatively, whether to perform the first process may be determined based on the shooting mode, such as a moving image mode or a still image mode.

[0086] The image processing device may also determine whether to perform the first processing based on the PSF or OTF. For example, if the magnitude (spread) of the PSF is larger than a predetermined value, the first processing is performed, and if it is smaller, the second processing is performed. If the PSF is large, the first processing is performed, which allows recovery processing even with an image recovery filter with a small number of taps, and if the PSF is small, the second processing is performed. The magnitude of the PSF may be the half-width of the PSF, or may be determined as a range that includes a predetermined proportion of the PSF (for example, 95% of the total), etc.

[0087] Alternatively, the image processing device may determine whether to perform the first processing based on the OTF of a frequency equal to or higher than the Nyquist frequency. For example, when the MTF is smaller than a threshold value at a predetermined frequency equal to or higher than the Nyquist frequency, the first processing is performed, and when the MTF is larger than a threshold value at the predetermined frequency, the second processing is performed. For example, the predetermined frequency equal to or higher than the Nyquist frequency is the Nyquist frequency, and whether to perform the first processing is determined based on the MTF at the Nyquist frequency. As a result, when there is an MTF equal to or higher than the Nyquist frequency, the image restoration processing is performed after the high pixel processing, so that an appropriate image restoration processing can be performed for the high frequency generated by the high pixel processing. On the other hand, when there is little MTF equal to or higher than the Nyquist frequency, the first processing can be performed to perform an efficient image restoration processing with reduced data amount and calculation amount.

[0088] The image processing device may also determine whether to perform the first processing based on the aperture value. For example, the image processing device performs the first processing when the aperture value is larger than a predetermined aperture value, and performs the second processing when the aperture value is smaller than the predetermined aperture value. When the aperture value increases, the diffraction blur increases and the MTF of high frequencies also tends to decrease. Therefore, when the aperture value is large, the first processing is performed, which allows recovery processing even with an image recovery filter with a small number of taps, and when the aperture value is small, the second processing is performed, which allows appropriate image recovery processing for high frequencies generated by high pixel processing. This type of processing determination method is preferably adopted, for example, when the aberration is smaller than a predetermined amount. Note that when the aperture value is small, the influence of the aberration becomes dominant and the PSF tends to become large. For this reason, the image processing device may perform the first processing when the aperture value is smaller than a predetermined aperture value, and perform the second processing when the aperture value is larger than the predetermined aperture value. This type of processing determination method is preferably adopted when the aberration is larger than a predetermined amount.

[0089] Next, in step S22, the image processing device performs pixel-enhancing processing using the multiple captured images acquired in step S11. As described above, various techniques can be used for pixel-enhancing processing.

[0090] Next, in step S23, the image processing device acquires an image restoration filter to be used in the image restoration process described later. In this embodiment, similarly to the first embodiment, aberration information is acquired based on the shooting conditions, and an image restoration filter is acquired based on the aberration information, but a pre-stored image restoration filter may be acquired. Here, the image restoration filter acquired in the second process is an image restoration filter that also restores frequencies equal to or higher than the Nyquist frequency generated by the high pixelation process, and is different from the image restoration filter acquired in step S12 of the first process. In addition, the image restoration filter acquired in the first process has a smaller number of taps than the image restoration filter acquired in the second process.

[0091] Next, in step S24, the image processing device performs image restoration processing on the image obtained by performing the high pixel processing in step S22 using the image restoration filter obtained in step S23, thereby obtaining a high pixel image that has been image restored. Next, in step S25, the image processing device outputs the high pixel image that has been image restored as an output image. At this time, the image processing device may perform various processes related to development processing on the high pixel image.

[0092] The high-pixel image obtained by the above flow is an image in which blurring due to aberration and diffraction is reduced by image restoration processing, and the number of pixels is greater than the number of pixels of the image sensor. In addition, in this embodiment, by appropriately determining whether to perform the first processing or the second processing depending on various conditions, it is possible to perform processing that is effective for various conditions. EXAMPLES

[0093] Next, an image processing system including an image processing device that performs the above-mentioned image processing method will be described with reference to Fig. 12. Fig. 12 is an explanatory diagram of an image processing system 300 in this embodiment. The image processing system 300 includes an aberration information calculation device 301, a camera (imaging device) 310, and an image processing device 320.

[0094] The aberration information calculation device 301 performs a process of calculating an optical transfer function OTF from design values ​​or measured values ​​of the imaging optical system according to the imaging conditions of the captured image. The camera 310 has an imaging element 311 and an imaging lens 312. The camera 310 adds the lens ID of the imaging lens 312, imaging condition information (aperture value, zoom, shooting distance, etc.), and the Nyquist frequency of the imaging element 311 to an image captured by the imaging lens 312, and outputs the image.

[0095] The image processing device 320 includes an image restoration information holding unit 321, a captured image acquisition unit (image acquisition unit) 322, a restoration filter acquisition unit 323, an image restoration processing unit 324, and a pixel enhancement processing unit 325. When the image processing device 320 performs processing according to the flow of the second embodiment, it further includes a processing determination unit. The image processing device 320 holds information output from the aberration information calculation device 301 and the camera 310, and uses this information to correct a degraded image captured by the imaging lens 312 (performs image restoration processing of the captured image).

[0096] The image restoration information storage unit 321 stores information on the optical transfer function OTF, the number of taps, the lens ID, the shooting conditions, and the Nyquist frequency of the image sensor for each of the various combinations of the imaging lens 312 and the image sensor 311 calculated by the aberration information calculation device 301. In this way, the image restoration information storage unit 321 is a storage unit that stores the optical transfer function OTF according to the shooting conditions of the captured image. Alternatively, instead of the optical transfer function OTF, an image restoration filter may be generated and stored. The captured image acquisition unit 322 acquires the captured image from the camera 310.

[0097] The restoration filter acquisition unit 323 acquires Nyquist frequency information of the imaging element of the imaging element 311 and the captured image from the camera 310, and acquires the lens ID and imaging condition information of the imaging lens 312. The restoration filter acquisition unit 323 searches for an optical transfer function OTF stored in the image restoration information storage unit 321 based on the lens ID and imaging conditions of the camera 310 used by the photographer when capturing an image. Then, the restoration filter acquisition unit 323 acquires a corresponding optical transfer function OTF (an optical transfer function OTF appropriate for the lens ID and imaging conditions at the time of capturing an image). The restoration filter acquisition unit 323 acquires an optical transfer function OTF used by the restoration filter acquisition unit 323 in a spatial frequency domain up to the Nyquist frequency of the imaging element of the camera 310. That is, the restoration filter acquisition unit 323 acquires an optical transfer function OTF of the imaging optical system (imaging lens 312) according to the position of the captured image using the acquired optical transfer function OTF. In this way, the restoration filter acquisition unit 323 is an OTF acquisition unit that acquires the optical transfer function OTF of the imaging optical system according to the position of the captured image. The restoration filter acquisition unit 323 is also an OTF expansion unit that expands the optical transfer function OTF by rotating the optical transfer function OTF around the center of the captured image or the optical axis OA of the imaging optical system. Furthermore, the restoration filter acquisition unit 323 generates an image restoration filter that corrects deterioration of the captured image using the acquired optical transfer function OTF. Note that, if an image restoration filter is held in the image restoration information holding unit 321, the restoration filter acquisition unit 323 acquires the image restoration filter.

[0098] The image restoration processing unit 324 corrects image degradation using the acquired image restoration filter. The pixel enhancement processing unit 325 performs pixel enhancement processing using a plurality of images captured at mutually shifted positions to acquire a pixel enhancement image.

[0099] If the optical transfer function OTF calculated in advance by the aberration information calculation device 301 is stored in the image restoration information storage unit 321, there is no need to provide the aberration information calculation device 301 to the user (photographer). In addition, the user can download and use information necessary for image restoration processing, such as coefficient data, via a network or various storage media. EXAMPLES

[0100] Next, an imaging device in a fourth embodiment of the present invention will be described with reference to Fig. 13. Fig. 13 is a block diagram of an imaging device 400 in this embodiment. An image processing program that performs image restoration processing of a captured image (an image processing method similar to that of the first or second embodiment) is installed in the imaging device 400, and this image restoration processing is executed by an image processing unit 404 (image processing device) inside the imaging device 400.

[0101] The imaging device 400 is configured with an imaging optical system 401 (lens) and an imaging device body (camera body). The imaging optical system 401 includes an aperture 401a and a focus lens 401b, and is configured integrally with the imaging device body (camera body). However, this embodiment is not limited to this, and can also be applied to an imaging device in which the imaging optical system 401 is replaceably attached to the imaging device body. The imaging optical system 401 may also include a phase mask (wavefront modulation element) that reduces performance fluctuation in the depth direction.

[0102] The image sensor 402 photoelectrically converts the subject image (optical image, imaging light) formed via the imaging optical system 401 to generate a captured image. That is, the subject image is photoelectrically converted by the image sensor 402 and converted into an analog signal (electrical signal). This analog signal is then converted into a digital signal by the A / D converter 403, and this digital signal is input to the image processing unit 404.

[0103] The image processing unit (image processing device) 404 performs a predetermined process on the digital signal and also performs the image restoration process described above. The image processing unit 404 has a captured image acquisition unit (image acquisition unit) 404a, a restoration filter acquisition unit 404b, an image restoration processing unit 404c, and a pixel enhancement processing unit 404d. When the image processing unit 404 performs the process according to the flow of the second embodiment, it further has a process determination unit 404e.

[0104] First, the image processing unit 404 acquires imaging condition information of the imaging device from the state detection unit 407. The imaging condition information is information related to the aperture value (F value), shooting distance, or focal length of the zoom lens. The state detection unit 407 can acquire the imaging condition information directly from the system controller 410, but is not limited to this. For example, the imaging condition information related to the imaging optical system 401 can also be acquired from the imaging optical system control unit 406. The processing flow (image processing method) of the image restoration processing of this embodiment is similar to that of the first embodiment described with reference to FIG. 7, and therefore the description thereof will be omitted.

[0105] The optical transfer function OTF or the image restoration filter is stored in a storage unit 408. The output image processed by the image processing unit 404 is stored in a predetermined format in an image recording medium 409. An image obtained by performing a predetermined process for display on an image that has been subjected to the image restoration process of this embodiment is displayed on the display unit 405. However, the present invention is not limited to this, and the display unit 405 may be configured to display an image that has been subjected to simple processing for high-speed display.

[0106] A series of controls in this embodiment are performed by the system controller 410, and the mechanical drive of the imaging optical system 401 is performed by the imaging optical system control unit 406 based on instructions from the system controller 410. The imaging optical system control unit 406 controls the aperture diameter of the diaphragm 401a as a shooting state setting of the aperture value (F value). The imaging optical system control unit 406 also controls the position of the focus lens 401b by an autofocus (AF) mechanism or a manual focus mechanism (not shown) to adjust the focus according to the subject distance. Note that the functions such as the aperture diameter control of the diaphragm 401a and manual focus may not be executed depending on the specifications of the imaging device 400.

[0107] The imaging optical system 401 may include optical elements such as a low-pass filter and an infrared cut filter, but when using an element such as a low-pass filter that affects the characteristics of the optical transfer function OTF, consideration may be required when creating an image restoration filter. The infrared cut filter also affects each PSF of the RGB channels, which is the integral value of the point spread function (PSF) of the spectral wavelength, especially the PSF of the R channel, so consideration may be required when creating an image restoration filter. In this case, as described in the first embodiment, a non-rotationally symmetric transfer function is added after rearranging the optical transfer function OTF.

[0108] In this embodiment, the optical transfer function OTF and the image restoration filter stored in the memory unit 408 of the imaging device are used, but as a variant, the imaging device may be configured to acquire the optical transfer function OTF and the image restoration filter stored in a storage medium such as a memory card.

[0109] Thus, in each embodiment, the image processing device 320 (image processing unit 404) has a photographed image acquisition unit 322 (404a), an image restoration processing unit 324 (404c), and a pixel-enhancing processing unit 325 (404d). The photographed image acquisition unit acquires a first image and a second image (a plurality of photographed images) having different imaging positions. The image restoration processing unit performs image restoration processing on the first image and the second image, and acquires a first restored image and a second restored image, respectively. The pixel-enhancing processing unit acquires a pixel-enhancing image using the first restored image and the second restored image. Preferably, the pixel-enhancing image has a larger number of pixels than each of the first image and the second image. Also preferably, the image processing device has a restoration filter acquisition unit 323 (404b) that acquires an image restoration filter. The image restoration processing unit performs image restoration processing on the first image and the second image using the image restoration filter.

[0110] Preferably, the image processing device has a processing determination unit that determines whether to perform the first processing or the second processing. When the processing determination unit determines to perform the first processing, the image restoration processing unit performs image restoration processing on the first image and the second image to obtain the first restored image and the second restored image, and the pixel enhancement processing unit obtains a pixel-enhanced image using the first restored image and the second restored image. On the other hand, when the processing determination unit determines to perform the second processing, the pixel enhancement processing unit obtains a pixel-enhanced image using the first image and the second image, and the image restoration processing unit performs image restoration processing on the pixel-enhanced image to obtain a pixel-enhanced image that has been restored.

[0111] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0112] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, and a program that can reduce the amount of data and the amount of calculations while reducing blurring due to aberrations and diffraction in the optical system when performing high-pixel processing and image restoration processing.

[0113] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0114] 320 Image Processing Device 322 Photographed image acquisition unit (image acquisition unit) 324 Image recovery processing section 325 High pixel processing unit

Claims

1. an image acquisition unit that acquires a plurality of images captured at different positions; a first processing unit that performs a restoration process on the plurality of images to obtain a plurality of restored images; a second processing unit that acquires a third image using the plurality of restored images; the first processing unit performs a recovery process on the plurality of images after all of the plurality of images are acquired by the image acquisition unit; 13. An image processing apparatus comprising: a third image having a number of pixels greater than the number of pixels of each of the plurality of images;

2. The image processing apparatus according to claim 1 , wherein the resolution of the third image is higher than the resolution of each of the plurality of images.

3. A restoration filter acquisition unit for acquiring an image restoration filter is further provided. 3 . The image processing apparatus according to claim 1 , wherein the first processing unit performs the restoration process on the images by using the image restoration filter. 4 .

4. 4. The image processing apparatus according to claim 3, wherein the image restoration filter is generated based on an optical transfer function at frequencies equal to or lower than the Nyquist frequency and an optical transfer function at frequencies higher than the Nyquist frequency.

5. 4. The image processing apparatus according to claim 3, wherein the image restoration filter is generated based on restoration characteristics at frequencies equal to or lower than the Nyquist frequency and restoration characteristics at frequencies higher than the Nyquist frequency.

6. 6. The image processing apparatus according to claim 3, wherein the image restoration filter is generated based on optical information of an image pickup optical system.

7. Further comprising a process determination unit that determines whether the first process or the second process is to be performed, When the processing determination unit determines that the first processing is to be performed, The first processing unit performs a recovery process on the plurality of images to obtain the plurality of recovered images; The second processing unit acquires the third image using the plurality of restored images; When the processing determination unit determines that the second processing is to be performed, The second processing unit acquires the third image using the plurality of images, The image processing apparatus according to claim 1 , wherein the first processing unit acquires a fourth image by performing a restoration process on the third image.

8. The processing determination unit determining whether to perform the first process or the second process based on recovery target data including the plurality of images; If the recovery target data is a video, it is determined that the first process is to be performed; 8. The image processing apparatus according to claim 7, wherein when the data to be recovered is a still image, it is determined that the second process is to be performed.

9. 8. The image processing apparatus according to claim 7, wherein the processing determination unit determines whether the first processing or the second processing is to be performed based on a point spread function or an optical transfer function of an imaging optical system.

10. The processing determination unit If the point spread function is greater than a predetermined value, it is determined that the first processing is to be performed; 10. The image processing apparatus according to claim 9, wherein, when the point spread function is smaller than the predetermined value, it is determined that the second processing is to be performed.

11. The processing determination unit When the absolute value of the optical transfer function is smaller than a threshold value at a predetermined frequency, it is determined that the first processing is to be performed; 10. The image processing apparatus according to claim 9, wherein, when an absolute value of the optical transfer function is greater than the threshold value at the predetermined frequency, it is determined that the second process is to be performed.

12. The image processing apparatus according to claim 7 , wherein the process determination unit determines whether the first process or the second process is to be performed based on an aperture value of an imaging optical system.

13. The processing determination unit If the aperture value is greater than a predetermined aperture value, it is determined that the first process is to be performed; 13. The image processing apparatus according to claim 12, wherein, when the aperture value is smaller than the predetermined aperture value, it is determined that the second process is to be performed.

14. The processing determination unit If the aperture value is smaller than a predetermined aperture value, it is determined that the first process is to be performed; 13. The image processing apparatus according to claim 12, wherein, when the aperture value is larger than the predetermined aperture value, it is determined that the second process is to be performed.

15. 15. The image processing apparatus according to claim 7, wherein the number of taps of an image restoration filter used in the restoration process in the first process and the number of taps of an image restoration filter used in the restoration process in the second process are different from each other.

16. An image acquisition unit that acquires a first image and a second image having different imaging positions; a first processing unit that performs a recovery process on the first image and the second image to obtain a first recovered image and a second recovered image, respectively; a second processing unit that acquires a third image using the first restored image and the second restored image; a process determination unit that determines whether the first process or the second process is to be performed; When the processing determination unit determines that the first processing is to be performed, the first processing unit performs a recovery process on the first image and the second image to obtain the first recovered image and the second recovered image, and the second processing unit uses the first recovered image and the second recovered image to obtain the third image, When the processing determination unit determines that the second processing is to be performed, the second processing unit obtains the third image by using the first image and the second image, and the first processing unit obtains a fourth image by performing a recovery process on the third image.

13. An image processing apparatus comprising: a third image having a larger number of pixels than each of the first image and the second image.

17. an imaging element that photoelectrically converts an optical image formed by the imaging optical system; An imaging device comprising: an image processing device according to claim 1 .

18. The imaging device of claim 17 , wherein the imaging optics includes a phase mask.

19. A first step of acquiring a plurality of images at different imaging positions; a second step of performing a restoration process on the plurality of images to obtain a plurality of restored images; and a third step of acquiring a third image based on the plurality of restored images. The second step is executed after the first step is executed, An image processing method, comprising: a number of pixels of the third image that is greater than the number of pixels of each of the plurality of images.

20. A program causing a computer to execute the image processing method according to claim 19.

Images