IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND STORAGE MEDIUM

Abstract

To provide an image processing apparatus capable obtaining an image effect obtained by an aberration, even when an aperture is small.SOLUTION: An image processing apparatus (320) comprises: acquisition units (321, 322) for acquiring a captured image obtained by using an optical system and information on an image-capturing condition corresponding to the captured image; and a processing unit (324) which on the basis of an aperture value of an optical system included in the information on the image-capturing condition, executes deterioration processing on the captured image. A first deterioration amount in the deterioration processing when the aperture value is a first aperture value is smaller than a second deterioration amount in the deterioration processing when the aperture value is a second aperture value larger than the first aperture value.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, a program, and a storage medium. [Background technology]

[0002] Conventionally, imaging optical systems capable of expressing not only resolution but also favorable appearance (image effects such as soft focus effect) by utilizing the effects of aberrations such as spherical aberration have been known. The soft focus effect, for example, enables depiction of light bleeding from highlighted areas and depiction of smoothing fine structures of skin by intentionally generating spherical aberration. Patent Document 1 discloses an imaging optical system capable of obtaining a soft focus effect in which the main subject is softly blurred by controlling the amount of spherical aberration. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-47297 Summary of the Invention [Problem to be solved by the invention]

[0004] In the imaging optical system disclosed in Patent Document 1, when the aperture of the diaphragm is small (the aperture value is large), the effect of spherical aberration is small, and therefore the image effect (soft focus effect) is reduced.

[0005] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to provide an image processing device that can obtain an image effect using aberration even when the aperture of the diaphragm is small. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an image processing device, comprising: an acquisition unit that acquires a photographed image obtained using an optical system and information regarding photographing conditions corresponding to the photographed image; and based on an aperture value of the optical system included in the information regarding the photographing conditions, a degradation filter acquisition unit for acquiring a degradation filter; a processing unit that performs degradation processing on the captured image, Filtering the captured image The first deterioration amount is the deterioration amount when the aperture value is a second aperture value that is larger than the first aperture value. Filtering the captured image The deterioration amount is smaller than the second deterioration amount.

[0007] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device capable of obtaining an image effect using aberration even when the aperture of the diaphragm is small. [Brief description of the drawings]

[0009] [Figure 1] 5 is an explanatory diagram of a degradation filter in each embodiment. FIG. [Diagram 2] 4 is an explanatory diagram (sectional view) of a deterioration filter in each embodiment. FIG. [Diagram 3] 5A and 5B are explanatory diagrams of point spread functions used as degradation filters in the respective embodiments. [Figure 4] 10A and 10B are explanatory diagrams of amplitude components of an optical transfer function that is a Fourier transform of a point spread function used as a degradation filter in each embodiment. [Diagram 5] 4 is a flowchart of an image processing method in the first embodiment. [Figure 6] 3A and 3B are diagrams illustrating images captured by an imaging optical system having spherical aberration in Example 1. [Figure 7] 5 is a diagram showing the amount of deterioration caused by deterioration processing relative to the amount of aberration of the imaging optical system in the first embodiment. FIG. [Figure 8] 4 is a diagram showing a degraded image obtained by performing degradation processing on an image captured by the imaging optical system having spherical aberration in the first embodiment. FIG. [Figure 9] 4 is a diagram showing an image captured using an imaging optical system without spherical aberration in Example 1. FIG. [Figure 10] 4 is a diagram showing a degraded image obtained by performing degradation processing on an image captured by the imaging optical system having no spherical aberration in the first embodiment. FIG. [Figure 11] 10 is a flowchart of an image processing method according to a second embodiment. [Figure 12] FIG. 11 is a diagram illustrating a method for generating a degradation filter in the second embodiment. [Figure 13] FIG. 11 is an explanatory diagram of an image processing system according to a third embodiment. [Figure 14] FIG. 13 is a block diagram of an imaging device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] First, the definitions of terms used in each embodiment and degradation processing (image processing method) will be described. The image processing method described here will be used appropriately in each embodiment described later. [Photo] The captured image obtained by the imaging device is a digital image obtained by receiving light by an imaging element through an imaging optical system, and contains blur components and is degraded due to the influence of various aberrations such as spherical aberration, coma aberration, field curvature, astigmatism, etc. of the imaging optical system including lenses and various optical filters. The blur components of the image caused by such aberrations mean that the light beam emitted from one point of the subject should be converged again to one point on the imaging surface in the absence of aberration and the influence of diffraction, but the light beam spreads out. The blur components of the image are expressed by a point spread function (PSF) or an optical transfer function (OTF). The imaging optical system can be constructed using not only lenses but also mirrors (reflecting surfaces) having a curvature.

[0012] The color components of the captured image include, for example, information on RGB color components. As color components, other commonly used color spaces such as lightness, hue, and saturation expressed by LCH, and luminance and color difference signals expressed by YCbCr can be selected and used. Other color spaces that can be used include XYZ, Lab, Yuv, and JCh. Furthermore, color temperature may be used.

[0013] The captured image (input image) and output image can be accompanied by shooting conditions such as the focal length of the lens, the aperture value (F value), and the shooting distance, as well as various correction information for performing processing such as correction on the image. When an image is transferred from the imaging device to another image processing device for processing such as correction, it is preferable to attach information about the shooting conditions and processing such as correction to the captured image as described above. As another method of transferring information about the shooting conditions and processing such as correction, the imaging device and the image processing device may be directly or indirectly connected to transfer the information. In each embodiment, the image (captured image) to be subjected to degradation processing may be either a still image or a video image. [Aging process] Next, an overview of degradation processing will be explained. First, considering the process of capturing an image using an imaging optical system, the captured image can be expressed by the following formula (1), where g(x,y) is the captured image, f(x,y) is the original image, and h(x,y) is the point spread function PSF, which is a Fourier pair of the optical transfer function OTF.

[0014] g(x,y)=h(x,y)*f(x,y) … (1) Here, * is convolution (convolution integral, product sum), and (x, y) are coordinates on the captured image. h(x, y) is a characteristic based on the aberration of the imaging optical system. Therefore, h(x, y) changes depending on the aperture of the imaging optical system. In other words, when the aperture is reduced from the maximum aperture (increasing the aperture value), the effect of aberration decreases, and the amount of degradation of h(x, y) also tends to decrease.

[0015] Next, in the degradation process in which a degraded image g2 is obtained by performing a convolution process of a degradation filter d on a captured image g, the following formula (2) is established in a similar manner.

[0016] g2(x,y)=d(x,y)*g(x,y) … (2) Here, d(x,y) is called a degradation filter. When the image is a two-dimensional image, the degradation filter d is also a two-dimensional filter having taps (cells) corresponding to each pixel of the image. Generally, the larger the number of taps (number of cells) of the degradation filter d, the larger the spread due to degradation can be. For this reason, a feasible number of taps is set according to the required image quality, image processing capabilities, aberration or image effect characteristics (amount of degradation, spread), etc.

[0017] The degradation filter d preferably reflects the characteristics of the aberration, and is therefore different from a blurring filter having a simple Gaussian characteristic, etc. When the degradation filter d has characteristics equivalent to the point spread function PSF of an imaging optical system having spherical aberration, a soft focus effect can be imparted to the captured image.

[0018] Next, the degradation filter will be described with reference to Fig. 1 and Fig. 2. Fig. 1 and Fig. 2 are explanatory diagrams of the degradation filter. Fig. 1 shows a two-dimensional filter with 11 × 11 taps as an example. In Fig. 1, the values ​​(coefficients) in each tap are omitted, but Fig. 2 shows a cross section of the degradation filter. The distribution of the values ​​(coefficient values) of each tap of the degradation filter has the function of degrading a captured image with spatially spread signal values.

[0019] Each tap of the degradation filter is convolution-processed (convolution integral, product-sum) in the degradation processing step corresponding to each pixel of the image. In the convolution processing, in order to degrade the signal value of a specific pixel, the pixel is aligned with the center of the degradation filter. Then, for each corresponding pixel of the image and the degradation filter, the product of the image signal value and the filter coefficient value is calculated, and the sum of these products is replaced as the signal value of the center pixel.

[0020] Next, the characteristics of the degradation filter in real space and frequency space will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is an explanatory diagram of a point spread function PSF used as a degradation filter. Fig. 4 is an explanatory diagram of an amplitude component MTF of an optical transfer function OTF, which is a Fourier transform of the point spread function PSF used as a degradation filter. In Fig. 4, the horizontal axis indicates the spatial frequency, and the vertical axis indicates the amplitude component MTF.

[0021] As shown in Fig. 3, in the case of an imaging optical system having spherical aberration, the point spread function PSF has a distribution in which the central part has a large value and the peripheral part has a large small value, and this distribution has a soft filter effect. In other words, when the point spread function PSF of an imaging optical system having spherical aberration is used as a degradation filter, the distribution has more pixels that are equal to or greater than the effective value and smaller than the threshold than pixels that are equal to or greater than the threshold. In this case, if the threshold is set to half the maximum value of the degradation filter and the effective value is set to the reciprocal of the maximum value that the image can take, the distribution of the degradation filter has the above-mentioned soft filter effect.

[0022] In this way, the degradation filter can be obtained based on the point spread function PSF (or optical transfer function OTF) of the imaging optical system. However, the degradation filter used in this embodiment can be changed as appropriate. For example, the point spread function PSF of the imaging optical system may be adjusted and used as the degradation filter, or a point spread function PSF of the imaging optical system different from the point spread function PSF of the imaging optical system used for shooting may be used as the degradation filter. In addition, a unique degradation filter that is not derived from the aberration of the imaging optical system and that can obtain a desired image effect may be created and used.

[0023] In addition, since the point spread function PSF and the optical transfer function OTF due to aberration change according to the image height (image position) of the imaging optical system even in one shooting state, the degradation filter may be changed according to the image height. Alternatively, a uniform (fixed) degradation filter may be used within one image. For example, a degradation filter equivalent to the point spread function PSF on the optical axis may be used uniformly within the screen.

[0024] Regarding wavelength, the optical transfer function OTF or point spread function PSF can be calculated for multiple wavelengths, and the optical transfer function OTF or point spread function PSF for each color component can be generated by weighting for each wavelength based on the spectrum of the assumed light source and the light receiving sensitivity information of the image sensor. Alternatively, calculations may be performed at a representative wavelength for each predetermined color component. Then, a degradation filter can be generated based on the optical transfer function OTF or point spread function PSF for each color component.

[0025] As described above, the aberration of the imaging optical system decreases as the aperture value increases. For example, in an imaging optical system having spherical aberration and a soft focus effect, the soft focus effect decreases as the aperture value increases. Therefore, by storing a plurality of deterioration filters according to the aperture value in advance and performing deterioration processing using the deterioration filter according to the shooting condition of the aperture value, an image having a soft focus effect can be obtained even if the aperture is reduced.

[0026] Each embodiment will be described in detail below. EXAMPLES

[0027] First, an image processing method in the first embodiment of the present invention will be described with reference to Fig. 5. Fig. 5 is a flowchart of the image processing method (image processing program) in this embodiment. The image processing method in this embodiment is executed by a computer constituted by a CPU and the like as an image processing device in accordance with an image processing program as a computer program. This is the same in other embodiments described later.

[0028] First, in step S11, the image processing device acquires a captured image generated by the imaging device through imaging. The captured image may be acquired from the imaging device through wired or wireless communication between the imaging device and the image processing device, or through a storage medium such as a semiconductor memory or an optical disk.

[0029] Next, in step S12, the image processing device acquires the shooting conditions (shooting condition information) when the imaging device generates the captured image by imaging. As described above, the shooting conditions include the focal length, aperture value (F value), and shooting distance of the imaging optical system, as well as identification information (camera ID) of the imaging device. In addition, in an imaging device in which the imaging optical system can be replaced, the shooting conditions may include identification information (lens ID) of the imaging optical system (interchangeable lens). The shooting condition information may be acquired as information attached to the captured image as described above, or may be acquired via wired or wireless communication or a storage medium.

[0030] Next, in step S13, the image processing device acquires a degradation filter used in degradation processing described later. In this embodiment, the degradation filter is acquired based on the point spread function PSF of the imaging optical system. More specifically, in this embodiment, the degradation filter is acquired based on the point spread function PSF of the imaging optical system at the focal plane on the optical axis at the open aperture. However, the present invention is not limited to this.

[0031] The image processing device selects and acquires an appropriate degradation filter according to the shooting conditions from a plurality of degradation filters stored in advance for each shooting condition. In addition, when the shooting conditions such as the aperture value, shooting distance, and focal length of the zoom lens are specific shooting conditions, the degradation filter corresponding to the shooting condition can be generated by an interpolation process from degradation filters of other shooting conditions stored in advance. In this case, it is possible to reduce the amount of data of the degradation filters stored. For example, bilinear interpolation (linear interpolation) and bicubic interpolation are used as the interpolation process, but the present invention is not limited to these.

[0032] In this embodiment, a degradation filter is stored for each aperture value, and a degradation filter is acquired based on the aperture value. Therefore, in step S12, the image processing device needs to acquire at least the aperture value as a shooting condition.

[0033] Also, instead of the point spread function PSF used as the degradation filter, degradation information such as an optical transfer function OTF may be acquired, or a degradation filter based on the point spread function PSF obtained by Fourier transforming the optical transfer function OTF may be acquired. Alternatively, a degradation filter having frequency characteristics based on the optical transfer function OTF may be acquired, and a convolution process may be performed by multiplying a spectral image obtained by Fourier transforming the captured image and the degradation filter having frequency characteristics in the degradation process described later.

[0034] Alternatively, the degradation information of the imaging optical system or the degradation filter may be fitted to a predetermined function to obtain approximate coefficient data, and the optical transfer function OTF, the point spread function PSF, or the degradation filter may be reconstructed from the coefficient data to obtain the degradation filter. For example, the optical transfer function OTF may be fitted using the Legendre polynomial. Alternatively, fitting may be performed using other functions such as the Chebushev polynomial.

[0035] Also, a part of a two-dimensional degradation filter may be held and used. For example, the amount of data can be reduced by holding a degradation filter for one quadrant, which is 1 / 4 of the two-dimensional degradation filter. Alternatively, the amount of data can be reduced by holding a plurality of one-dimensional tap data that are components of the two-dimensional degradation filter. For example, the amount of data can be reduced by holding one-dimensional data that is half of the data in the horizontal and vertical directions including the center of the two-dimensional degradation filter. Not limited to this, the amount of data can be reduced by holding data obtained by decomposing the degradation filter using fitting or the singular value decomposition theorem, on the premise that the degradation filter is symmetric (symmetric within the tap). In other words, by utilizing the fact that the degradation filter is symmetric (symmetric within the tap), data reduction is possible by holding a part of data or component data that has less data amount than the two-dimensional degradation filter.

[0036] Here, it is preferable that the deterioration filter to be acquired is a filter that reduces the change in image effect according to the aperture of the imaging optical system. Fig. 6(a) to (c) are diagrams showing captured images according to each aperture value when shooting is performed with an imaging optical system having spherical aberration. Fig. 6(a) shows the case of open aperture value A, Fig. 6(b) shows the case of aperture value B, and Fig. 6(c) shows the case of aperture value C (open aperture value A<aperture value B<aperture value C). For example, when shooting is performed with an imaging optical system having spherical aberration (or with large spherical aberration) as shown in Fig. 6(a) to (c), a captured image with a desired image effect (soft filter effect) is obtained with open aperture value A, but the image effect decreases as the aperture value increases. Therefore, a deterioration filter is acquired so that an image effect close to that of the open aperture value can be obtained even if the aperture value is increased by acquiring a deterioration filter according to the aperture value and performing deterioration processing. In other words, a deterioration filter is acquired so that the deterioration amount of the deterioration processing at the first aperture value is smaller than the deterioration amount of the deterioration processing at the second aperture value larger than the first aperture value. In other words, when the aperture value is a first aperture value, the amount of deterioration in the deterioration processing is a first deterioration amount, and when the aperture value is a second aperture value greater than the first deterioration amount, the amount of deterioration is a second deterioration amount greater than the first deterioration amount.

[0037] Here, the degradation amount indicates the degree to which an image is degraded by a degradation filter, and corresponds to the amount of blur when degradation processing (blurring processing) is performed on a captured image. For example, the degradation amount is 1-(absolute value of the degradation filter in frequency space). Alternatively, the degradation amount may be the average value of 1-(absolute value of the filter in frequency space), or the weighted average value of the values ​​at each frequency. Also, the spread width of the degradation filter may be the degradation amount.

[0038] The deterioration amount may be adjusted by reducing the deterioration filter (point spread function PSF at the open aperture value) that maximizes the deterioration amount, or by weighting a delta function with a center of 1 and other than the center of 0 for the deterioration filter. A deterioration filter with an adjusted deterioration amount may be obtained in advance, or the deterioration amount may be adjusted after the deterioration filter is obtained. Alternatively, the deterioration filter may be obtained by removing the components of the point spread function PSF at each aperture value of the imaging optical system from the deterioration filter (point spread function PSF at the open aperture value) that maximizes the deterioration amount. Specifically, the deterioration filter may be obtained by inverse Fourier transform of the optical transfer function OTF at the open aperture value divided by the optical transfer function OTF at each aperture value.

[0039] Fig. 7 is a diagram showing the deterioration amount of the deterioration process with respect to the amount of aberration of the imaging optical system. As shown in Fig. 7, for example, it is preferable to increase the deterioration amount of the deterioration filter according to the aperture value with respect to the amount of aberration with respect to the aperture value of the imaging optical system, so that the deterioration amount of the deteriorated image, that is, the image effect does not change with the aperture value. Alternatively, it is preferable to increase the deterioration amount of the deterioration filter according to the aperture value with respect to the amount of aberration with respect to the aperture value of the imaging optical system, so that the change in the image effect according to the aperture of the imaging optical system becomes small.

[0040] 8(a) to 8(c) are diagrams showing degraded images obtained by performing degradation processing on each captured image corresponding to each aperture value when capturing images using an imaging optical system having spherical aberration. FIG. 8(a) shows the case of open aperture value A, FIG. 8(b) shows the case of aperture value B, and FIG. 8(c) shows the case of aperture value C (open aperture value A<aperture value B<aperture value C). In the captured images shown in FIG. 6(a) to 8(c), the image effect decreases as the aperture value increases. On the other hand, in the degraded images shown in FIG. 8(a) to 8(c), the image effect at the open aperture can be maintained even if the aperture value is increased. The degradation filter acquired here is preferably based on the point spread function PSF at the open aperture value of the imaging optical system, and a degradation filter may be stored for each imaging optical system and acquired according to the imaging optical system.

[0041] In addition, when degradation processing is performed on an image captured using an imaging optical system capable of adjusting the amount of spherical aberration, a degradation filter may be acquired according to the amount of aberration of the imaging optical system. In other words, the degradation amount of the degradation filter may be increased as the amount of aberration of the imaging optical system increases.

[0042] 9(a) to (c) are diagrams showing captured images corresponding to each aperture value when capturing images using an imaging optical system without spherical aberration. FIG. 9(a) shows the case of open aperture value A, FIG. 9(b) shows the case of aperture value B, and FIG. 9(c) shows the case of aperture value C (open aperture value A<aperture value B<aperture value C). As shown in FIG. 9(a) to (c), when capturing images using an imaging optical system without spherical aberration (or with small spherical aberration), an image effect like that of an imaging optical system with spherical aberration cannot be obtained. Therefore, by acquiring a degradation filter based on a point spread function PSF at the open aperture value of an imaging optical system with spherical aberration and performing degradation processing on the captured image, an image effect like that of an imaging optical system with spherical aberration can be obtained even in an imaging optical system without spherical aberration. In other words, a degradation filter based on a point spread function PSF of an imaging optical system different from the imaging optical system used when capturing the captured image may be acquired.

[0043] In this case, similarly, a degradation filter is acquired so that degradation processing is performed using a degradation filter according to the aperture value, and an image effect close to the open aperture value of an imaging optical system having spherical aberration can be obtained even if the aperture value is increased. Since the open aperture value of an imaging optical system without spherical aberration has a certain degree of influence of aberration, a degradation filter with a small degradation amount is acquired near the open aperture value, taking the amount of aberration into consideration. In other words, a degradation filter is acquired so that the degradation amount of degradation processing at a first aperture value is smaller than the degradation amount of degradation processing at a second aperture value larger than the first aperture value.

[0044] 10(a) to (c) are diagrams showing degraded images obtained by performing degradation processing on captured images corresponding to each aperture value when capturing images using an imaging optical system without spherical aberration. FIG. 10(a) shows the case of open aperture value A, FIG. 10(b) shows the case of aperture value B, and FIG. 10(c) shows the case of aperture value C (open aperture value A<aperture value B<aperture value C). The captured images shown in FIG. 9(a) to (c) do not provide an image effect equivalent to an imaging optical system having spherical aberration. On the other hand, the degraded images shown in FIG. 10(a) to (c) not only provide the image effect of an imaging optical system having spherical aberration, but also maintain the image effect due to aberration even when the aperture value increases.

[0045] In this embodiment, the point spread function PSF of the imaging optical system having spherical aberration is used as the degradation filter, but the present invention is not limited to this, and degradation filters with various characteristics can be used. For example, the point spread function PSF of the imaging optical system having coma aberration may be used as the degradation filter, or a degradation filter that can obtain an effect similar to the image effect due to the aberration of the imaging optical system may be created and used independently.

[0046] Next, in step S14 of Fig. 5, the image processing device performs degradation processing on the captured image using a degradation filter. That is, a convolution process is performed on the captured image using the degradation filter. When a degradation filter based on a point spread function PSF on the optical axis is acquired as in this embodiment, a uniform (fixed) degradation filter is used for the position of the image. Also, similar to the point spread function PSF of the imaging optical system that changes depending on the screen position, when a degradation filter is acquired for each screen position, the degradation filter may be changed and used depending on the screen position.

[0047] Next, in step S15, the image processing device acquires the degraded image that has been subjected to the degradation process. At this time, the image processing device may perform various processes related to development processing on the degraded image.

[0048] In this embodiment, the degradation filter is acquired and the captured image is subjected to a convolution process to acquire the degraded image, but the degradation process may be performed using other methods. For example, the degradation process may be performed by reducing and enlarging the captured image, or the degradation process may be performed by taking a weighted average of the captured image and the image obtained by reducing and enlarging the captured image. Alternatively, the degradation amount may be adjusted by taking a weighted average of the image convolved with the degradation filter and the captured image. Furthermore, the degradation amount may be adjusted by performing the convolution process of the degradation filter multiple times. However, it is preferable to adjust the degradation amount of the degradation process so that the change in the image effect according to the aperture of the imaging optical system is small. EXAMPLES

[0049] Next, an image processing method in the second embodiment of the present invention will be described. In this embodiment, a process for determining whether or not to perform degradation processing based on the shooting conditions (shooting condition information) is included. FIG. 11 is a flowchart of the image processing method (image processing program) in this embodiment. In this embodiment, a computer executes the image processing program according to the flow of FIG. 11. Note that steps S11, S12, S13, S14, and S15 in FIG. 11 are the same as those in FIG. 5 described in the first embodiment, and therefore their description will be omitted.

[0050] In step S21, the image processing device determines whether or not to perform degradation processing. If the image processing device determines that degradation processing is to be performed, the image processing device performs degradation processing based on the flow of the first embodiment. On the other hand, if the image processing device determines that degradation processing is not to be performed, the image processing device ends this flow.

[0051] The process of determining whether or not to perform deterioration processing is performed based on the shooting conditions. For example, when the aperture value is equal to or greater than a threshold value, deterioration processing is performed. The threshold value may be an aperture value at which the image effect due to the aberration of the imaging optical system is lost. Also, whether or not to perform deterioration processing may be determined based on information on the imaging optical system. For example, deterioration processing may be performed when the imaging optical system has spherical aberration, and deterioration processing may not be performed when the imaging optical system does not have spherical aberration. Thereby, while the image effect is reduced according to the aperture value when the imaging optical system has spherical aberration, by performing deterioration processing, it is possible to obtain the image effect due to the aberration of the imaging optical system even if the aperture value is increased. Alternatively, by performing deterioration processing when the imaging optical system does not have spherical aberration, it is possible to obtain an image as if it was taken by an imaging optical system having spherical aberration, even with an imaging optical system without spherical aberration. In this case, the user may select in advance whether or not to perform deterioration processing, and further determine whether or not to perform deterioration processing according to the processing mode.

[0052] Next, in step S13, the image processing device acquires a degradation filter used in degradation processing described later. In this embodiment, an example of acquiring a degradation filter for each position on the screen based on the shooting conditions will be described. In the embodiment, an optical transfer function OTF is acquired as degradation information, and a degradation filter is acquired based on the optical transfer function OTF.

[0053] First, the image processing device acquires degradation information suitable for the shooting conditions. In this embodiment, the degradation information is an optical transfer function OTF. The image processing device selects and acquires an appropriate optical transfer function OTF according to the shooting conditions from a plurality of optical transfer functions OTF stored in advance. In addition, when the shooting conditions such as the aperture value, shooting distance, and focal length of the zoom lens are specific shooting conditions, the optical transfer function OTF corresponding to the shooting condition can be generated by an interpolation process from the optical transfer functions OTF of other shooting conditions stored in advance. In this case, it is possible to reduce the amount of data of the optical transfer function OTF stored. For example, bilinear interpolation (linear interpolation) or bicubic interpolation is used as the interpolation process, but is not limited thereto. In this embodiment, the optical transfer function OTF at the open aperture value is acquired, and a degradation filter is acquired based on the acquired optical transfer function OTF.

[0054] In this embodiment, the image processing device acquires an optical transfer function OTF as degradation information, but is not limited to this. Instead of the optical transfer function OTF, degradation information such as a point spread function PSF may be acquired and used as a degradation filter. In this embodiment, the image processing device may acquire coefficient data that approximates the degradation information by fitting to a predetermined function, and reconstruct the optical transfer function OTF and the point spread function PSF from the coefficient data. For example, the optical transfer function OTF may be fitted using Legendre polynomials. Also, fitting can be performed using other functions such as Chebushev polynomials. In this embodiment, the image processing device generates multiple optical transfer functions OTF in one direction passing through the center of the screen (the center of the captured image) or the optical axis of the imaging optical system.

[0055] Next, the image processing device rotates the optical transfer function OTF around the screen center (center of the captured image) or the optical axis of the imaging optical system to expand the optical transfer function OTF. Specifically, the image processing device interpolates the optical transfer function OTF in accordance with the pixel array, thereby discretely disposing the optical transfer function OTF at multiple positions in the captured image.

[0056] Next, the image processing device converts the optical transfer function OTF into a degradation filter, i.e., generates a degradation filter using the expanded optical transfer function OTF. The degradation filter is generated by creating degradation filter characteristics in frequency space based on the optical transfer function OTF and converting it into a filter (degradation filter) in real space by inverse Fourier transform.

[0057] 12(a) to 12(e) are explanatory diagrams of a method for generating a degradation filter. As shown in Fig. 12(a), the optical transfer function OTF is arranged in a region of a circumscribing circle of an image (imaging region) in one direction (vertical direction) passing through the center of the screen (center of the captured image) or the optical axis of the imaging optical system.

[0058] In this embodiment, the optical transfer function is expanded on a straight line as shown in FIG. 12(a), but is not limited thereto. For example, in the captured image plane, the straight lines passing through the center of the captured image or the optical axis of the imaging optical system and perpendicular to each other are defined as a first straight line (y in FIG. 12(a)) and a second straight line (x in FIG. 12(a)). In this case, at least two of the acquired optical transfer functions may be optical transfer functions corresponding to the position (image height) on the first straight line. That is, the optical transfer functions OTF may not be linearly arranged in one direction as long as they are arranged at multiple positions (multiple positions in the captured image) arranged at different distances from the center of the screen or the optical axis of the imaging optical system in a predetermined direction. Note that when there is no pixel including the center of the captured image or the optical axis of the imaging optical system, that is, when the center of the captured image or the optical axis of the imaging optical system is between pixels, the acquired optical transfer functions may be optical transfer functions corresponding to the positions (image heights) of pixels sandwiching the first straight line.

[0059] In addition, when the optical transfer functions OTF are arranged in one direction, they are not limited to the vertical direction and may be arranged in other directions such as the horizontal direction. It is more preferable to arrange the optical transfer functions OTF linearly in either the vertical direction or the horizontal direction, since this embodiment of the image processing can be performed more easily.

[0060] Next, the optical transfer function OTF is rotated, and if necessary, an interpolation process (various processes according to the pixel arrangement after rotation) is performed to rearrange the optical transfer function OTF as shown in Fig. 12(b). The interpolation process includes an interpolation process in the radial direction and an interpolation process associated with the rotation, and the optical transfer function OTF can be rearranged to an arbitrary position.

[0061] Next, the optical transfer function OTF at each position is subjected to an inverse Fourier transform to convert it into a point spread function PSF in real space, that is, a degradation filter, as shown in FIG. 12(c).

[0062] That is, in the photographed image, the straight lines passing through the center of the photographed image or the optical axis of the imaging optical system and perpendicular to each other are defined as the first straight line (y in FIG. 12(a)) and the second straight line (x in FIG. 12(a)). The area that is point-symmetrical with the first area (83 in FIG. 12(c)) of the photographed image with respect to the center of the photographed image or the optical axis OA of the imaging optical system is defined as the second area (81 in FIG. 12(c)). In addition, the area that is line-symmetrical with respect to the first area and the first straight line is defined as the third area (82 in FIG. 12(c)), and the area that is line-symmetrical with respect to the first area and the second straight line is defined as the fourth area (84 in FIG. 12(c)). At this time, the optical transfer function of the first area is used to generate the optical transfer function of the second area, the third area, and the fourth area. This reduces the Fourier transform processing to approximately 1 / 4 of the position to be finally rearranged. Moreover, if the optical transfer function OTF in Fig. 12(b) and the degradation filter in Fig. 12(c) are rearranged by rotation and interpolation processing as shown in Fig. 12(e) and expanded as shown in Fig. 12(d) using the symmetry, the Fourier transform processing can be further reduced. Note that the arrangement (arrangement density of degradation filters) shown in Fig. 12(a) to (e) is one example, and the arrangement interval can be set arbitrarily according to the fluctuation of the optical transfer function OTF of the imaging optical system.

[0063] As described in the first embodiment, it is preferable that the degradation filter to be acquired is a filter that reduces the change in image effect according to the aperture of the imaging optical system. Therefore, the degradation filter is acquired so that the degradation process is performed by acquiring a degradation filter according to the aperture value based on the aperture value, so that an image effect close to that of the open aperture value can be obtained even if the aperture value is increased. In other words, the degradation filter is acquired so that the degradation amount of the degradation process at the first aperture value is smaller than the degradation amount of the degradation process at the second aperture value larger than the first aperture value. As in this embodiment, when acquiring a degradation filter from the optical transfer function OTF at the open aperture value of the imaging optical system, a degradation filter with an adjusted degradation amount may be acquired by inverse Fourier transforming the weighted average of the optical transfer function OTF and the characteristic of all frequencies being 1. Alternatively, the degradation amount may be reduced by shrinking the degradation filter (point spread function PSF at the open aperture value) with the maximum degradation amount, or the degradation amount may be reduced by weighting the delta function with the center being 1 and the rest being 0 for the degradation filter. In addition, the degradation amount may be changed depending on the aperture value by dividing the optical transfer function OTF at the open aperture value by the optical transfer function OTF at each aperture value.

[0064] In this embodiment, on the premise that the optical transfer function OTF is rotationally symmetrical with respect to the center of the imaging surface (center of the screen) or the optical axis of the imaging optical system, the optical transfer function OTF arranged in one direction passing through the center of the screen or the optical axis of the imaging optical system as described above is rotated and expanded. This makes it possible to obtain a degradation filter and perform degradation processing with a small amount of data.

[0065] Although the generation and acquisition of the degradation filter have been described above, the present invention is not limited thereto, and a degradation filter may be generated and stored in advance for each shooting condition, and the degradation filter may be similarly acquired based on the shooting condition. Also, the degradation filter may be acquired in a manner similar to step S13 described in the first embodiment. However, it is preferable that the acquired degradation filter is a filter that reduces the change in image effect according to the aperture of the imaging optical system.

[0066] Next, in step S14, the image processing device performs degradation processing on the captured image acquired in step S11. The degradation processing is performed based on the degradation filter acquired in step S13.

[0067] In addition, during the convolution of the degradation filter, pixels other than the position where the degradation filter shown in FIG. 12(d) is arranged can be generated by interpolation using a plurality of filters arranged in the vicinity. In this case, the degradation filter has a first degradation filter at a first position of the photographed image and a second degradation filter at a second position of the photographed image. The first degradation filter is generated by using the expanded optical transfer function. The second degradation filter is generated by interpolation using the first degradation filter. By performing such an interpolation process, it is possible to change the degradation filter for each pixel, for example. EXAMPLES

[0068] Next, an image processing system according to a third embodiment of the present invention will be described with reference to Fig. 13. The image processing system according to this embodiment includes an image processing device that performs the image processing method according to the first or second embodiment. Fig. 13 is an explanatory diagram of an image processing system 300 according to this embodiment. The image processing system 300 includes a degradation information calculation device 301, an imaging device 310, and an image processing device 320.

[0069] The degradation information calculation device 301 performs a process of calculating a point spread function PSF and an optical transfer function OTF to be used as degradation filters from design values ​​or measured values ​​of the imaging optical system according to the imaging conditions of the captured image. The imaging device 310 has an imaging element 311 and an imaging optical system 312. The imaging device 310 adds the lens ID and imaging condition information (aperture value, zoom, shooting distance, etc.) of the imaging optical system 312 and the Nyquist frequency of the imaging element 311 to the image captured by the imaging optical system 312, and outputs the image.

[0070] The image processing device 320 includes a degradation information holding unit 321, a captured image acquisition unit 322, a degradation filter acquisition unit 323, and a degradation processing unit 324. When the image processing device 320 performs processing according to the flow of the second embodiment, it has a separate processing determination unit. The image processing device 320 holds information output from the degradation information calculation device 301 and the imaging device 310, and uses this information to degrade the captured image captured by the imaging optical system 312 (performs degradation processing of the captured image).

[0071] The degradation information storage unit 321 stores a degradation filter or a point spread function PSF or an optical transfer function OTF for each of various combinations of the imaging optical system 312 and the imaging element 311 calculated by the degradation information calculation device 301. Furthermore, the degradation information storage unit 321 also stores information on the number of taps, the lens ID, the shooting conditions, and the Nyquist frequency of the imaging element. In this way, the degradation information storage unit 321 is a storage unit that stores a degradation filter according to the shooting conditions of the captured image. Alternatively, instead of a degradation filter, an optical transfer function OTF or a point spread function PSF for generating a degradation filter may be stored. Also, a degradation amount according to the shooting conditions may be stored, or the degradation amount of the degradation filter may be adjusted to be the degradation amount according to the shooting conditions.

[0072] The captured image acquisition unit 322 acquires a captured image from the imaging device 310. The degradation filter acquisition unit 323 acquires a captured image from the imaging device 310, and acquires imaging condition information including the lens ID of the imaging optical system 312 (information on the type of the imaging optical system 312). The degradation filter acquisition unit 323 searches for a degradation filter stored in the degradation information storage unit 321 based on the lens ID of the imaging device 310 used by the photographer when taking a picture and the imaging conditions. Then, the degradation filter acquisition unit 323 acquires a corresponding degradation filter (a degradation filter appropriate for the lens ID and the imaging conditions at the time of taking a picture). Here, it is preferable that the acquired degradation filter is a filter that reduces the change in image effect according to the aperture of the imaging optical system. In addition, a user may be allowed to select a virtual imaging optical system as the characteristic of the degradation filter, and a degradation filter may be acquired based on information on the virtual imaging optical system selected by the user.

[0073] In addition, when the optical transfer function OTF is held in the degradation information holding unit 321, the degradation filter acquisition unit 323 acquires a degradation filter based on the optical transfer function OTF. That is, the degradation filter acquisition unit 323 acquires a corresponding optical transfer function OTF (optical transfer function OTF appropriate for the lens ID and shooting conditions at the time of shooting). The degradation filter acquisition unit 323 acquires an optical transfer function OTF used by the degradation processing unit 324 in a spatial frequency domain up to the Nyquist frequency of the image sensor of the imaging device 310. That is, the degradation filter acquisition unit 323 acquires an optical transfer function OTF of the imaging optical system (imaging optical system 312) according to the position of the captured image using the acquired optical transfer function OTF. In this way, the degradation filter acquisition unit 323 is an OTF acquisition unit that acquires the optical transfer function OTF of the imaging optical system according to the position of the captured image. Furthermore, the degradation filter acquisition unit 323 generates a degradation filter that degrades the captured image using the acquired optical transfer function OTF. The same applies when a point spread function PSF is held instead of the optical transfer function OTF. The degradation processor 324 performs degradation processing on the captured image using the acquired degradation filter.

[0074] If a degradation filter calculated in advance by the degradation information calculation device 301 is stored in the degradation information storage unit 321, there is no need to provide the degradation information calculation device 301 to a user (photographer). In addition, a user can download and use information necessary for degradation processing, such as coefficient data, via a network or various storage media. EXAMPLES

[0075] Next, an imaging device according to a fourth embodiment of the present invention will be described with reference to Fig. 14. Fig. 14 is a block diagram of an imaging device 400 according to this embodiment. An image processing program that performs degradation processing of a captured image (the same image processing method as in the first or second embodiment) is installed in the imaging device 400, and this degradation processing is executed by an image processing unit (image processing device) 404 inside the imaging device 400.

[0076] The imaging device 400 is configured to include an imaging optical system (lens) 401 and an imaging device body (camera body). The imaging optical system 401 includes a diaphragm (aperture stop) 401a and a focus lens 401b, and is configured integrally with the imaging device body (camera body). However, this embodiment is not limited to this, and can also be applied to an imaging device in which the imaging optical system 401 is replaceably attached to the imaging device body.

[0077] The image sensor 402 photoelectrically converts the subject image (optical image, imaging light) formed via the imaging optical system 401 to generate a captured image. That is, the subject image is photoelectrically converted by the image sensor 402 and converted into an analog signal (electrical signal). The analog signal is then converted into a digital signal by the A / D converter 403, and the digital signal is input to the image processing unit 404.

[0078] The image processing unit 404 performs a predetermined process on the digital signal and also performs the degradation process described above. The image processing unit 404 has a captured image acquisition unit, a degradation filter acquisition unit, and a degradation processing unit, which are not shown. When the image processing unit 404 performs processing according to the flow of the second embodiment, it has a separate processing determination unit.

[0079] First, the image processing unit 404 has the same function as the image processing device 320 of the third embodiment, and acquires the photographing condition information of the imaging device from the state detection unit 407. The photographing condition information is information related to the aperture value (F value), the photographing distance, or the focal length of the zoom lens. The state detection unit 407 can acquire the photographing condition information directly from the system controller 410, but is not limited to this. For example, the photographing condition information related to the imaging optical system 401 can also be acquired from the imaging optical system control unit 406. The processing flow (image processing method) of the degradation processing of this embodiment is the same as that of the first embodiment described with reference to FIG. 5, and therefore the description thereof will be omitted. In this embodiment, the degradation filter is stored in the storage unit 408.

[0080] The output image processed by the image processing unit 404 is stored in a predetermined format in an image recording medium 409. An image obtained by performing predetermined processing for display on an image that has been subjected to the degradation processing of this embodiment is displayed on the display unit 405. However, the present invention is not limited to this, and the display unit 405 may be configured to display an image that has been subjected to simplified processing for high-speed display.

[0081] A series of controls in this embodiment are performed by the system controller 410, and the mechanical drive of the imaging optical system 401 is performed by the imaging optical system control unit 406 based on instructions from the system controller 410. The imaging optical system control unit 406 controls the aperture diameter of the diaphragm 401a as a shooting state setting of the aperture value (F value). The imaging optical system control unit 406 also controls the position of the focus lens 401b by an autofocus (AF) mechanism or a manual focus mechanism (not shown) to adjust the focus according to the subject distance. Note that the functions such as the aperture diameter control of the diaphragm 401a and manual focus may not be executed depending on the specifications of the imaging device 400.

[0082] In this embodiment, the deterioration filter stored in the storage unit 408 of the imaging device is used, but as a modified example, the imaging device may be configured to acquire a deterioration filter stored in a storage medium such as a memory card. Alternatively, the deterioration filter may be stored in a storage unit (not shown) built into the imaging optical system and acquired, or the deterioration filter may be copied from a storage unit built into the imaging optical system to the storage unit 408.

[0083] Thus, in each embodiment, the image processing device 320 (image processing unit 404) has a degradation information holding unit 321, a captured image acquisition unit 322, and a degradation processing unit 324. The degradation information holding unit 321 acquires and holds shooting condition information including the aperture value (F value) of the optical system (image capture optical system 312, 401). The captured image acquisition unit 322 acquires a captured image obtained through the optical system. That is, the captured image acquisition unit 322 and the degradation filter acquisition unit 323 function as an acquisition unit that acquires a captured image obtained using the optical system and information on the shooting condition corresponding to the captured image. The degradation processing unit 324 functions as a processing unit that performs degradation processing on the captured image based on the aperture value of the optical system included in the information on the shooting condition. A first degradation amount (a first blur amount when performing degradation processing (blur processing) on ​​the captured image) in degradation processing when the aperture value is a first aperture value is smaller than a second degradation amount (a second blur amount) in degradation processing when the aperture value is a second aperture value larger than the first aperture value.

[0084] Preferably, the processing unit performs degradation processing so as to reduce a difference between the amount of blur (deterioration) of the captured image due to aberration in the optical system when the aperture value is a first aperture value and the amount of blur (deterioration) of the captured image due to aberration in the optical system when the aperture value is a second aperture value. Also preferably, the processing unit performs degradation processing so as to reduce a difference between the amount of deterioration of the in-focus plane when the aperture value is a maximum aperture value and the amount of deterioration of the in-focus plane when the aperture value is a third aperture value larger than the maximum aperture value.

[0085] Preferably, the image processing device has a degradation filter acquisition unit 323 that acquires a degradation filter based on an aperture value. When the aperture value is a first aperture value, the degradation amount of the degradation filter is a first degradation amount, and when the aperture value is a second aperture value, the degradation amount of the degradation filter is a second degradation amount. More preferably, the shooting condition information includes information on the type of the optical system, and the degradation filter acquisition unit acquires the degradation filter based on the information on the type of the optical system. Also preferably, the degradation filter acquisition unit acquires the degradation filter based on a point spread function of a focal plane at the open aperture value of the optical system. Also preferably, the shooting condition information includes information on aberration of the optical system, and the degradation filter acquisition unit acquires the degradation filter based on the information on the aberration.

[0086] Preferably, the processing unit determines whether or not to perform deterioration processing based on shooting condition information. More preferably, the shooting condition information includes information on the type of optical system, and the processing unit determines whether or not to perform deterioration processing based on the information on the type of optical system. More preferably, the processing unit determines whether or not to perform deterioration processing based on an aperture value.

[0087] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0088] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium that are capable of obtaining an image effect due to aberration even when the aperture of the diaphragm is small.

[0089] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0090] 320 Image Processing Device 321 Deterioration information holding unit (acquisition unit) 322 Photographed image acquisition unit (acquisition unit) 324 Degradation Processing Unit (Processing Unit)

Claims

1. an acquisition unit that acquires a photographed image obtained by using an optical system and information regarding photographing conditions corresponding to the photographed image; a degradation filter acquisition unit that acquires a degradation filter based on an aperture value of the optical system included in the information about the photographing conditions; a processing unit that performs degradation processing on the captured image based on the degradation filter, An image processing device characterized in that a first amount of degradation of the captured image caused by the degradation filter when the aperture value is a first aperture value is smaller than a second amount of degradation of the captured image caused by the degradation filter when the aperture value is a second aperture value larger than the first aperture value.

2. 2 . The image processing apparatus according to claim 1 , wherein the first deterioration amount and the second deterioration amount are a first blur amount and a second blur amount given to the captured image, respectively.

3. The image processing device described in claim 1 or 2, characterized in that the processing unit performs the deterioration processing so as to reduce the difference between the amount of blur imparted to the captured image based on aberration in the optical system when the aperture value is the first aperture value and the amount of blur imparted to the captured image based on aberration in the optical system when the aperture value is the second aperture value.

4. The image processing device according to any one of claims 1 to 3, characterized in that the processing unit performs the deterioration processing so as to reduce the difference between the amount of blurring imparted to the captured image based on aberration in the optical system when the aperture value is an open aperture value and the amount of blurring imparted to the captured image based on aberration in the optical system when the aperture value is a third aperture value larger than the open aperture value.

5. the information on the photographing conditions includes information on the type of the optical system, 5 . The image processing apparatus according to claim 1 , wherein the degradation filter acquisition unit acquires the degradation filter on the basis of information relating to a type of the optical system.

6. The image processing apparatus according to claim 5 , wherein the degradation filter acquisition unit acquires the degradation filter based on a point spread function of a focal plane at a maximum aperture value of the optical system.

7. the information on the photographing conditions includes information on the aberration of the optical system, 7. The image processing apparatus according to claim 5, wherein the degradation filter acquisition unit acquires the degradation filter based on information about the aberration.

8. 8. The image processing apparatus according to claim 1, wherein the processing unit determines whether or not to perform the degradation process based on information about the shooting conditions.

9. the information on the photographing conditions includes information on the type of the optical system, The image processing apparatus according to claim 8 , wherein the processing unit determines whether or not to perform the deterioration process based on information about a type of the optical system.

10. The image processing device according to claim 8 , wherein the processing unit determines whether or not to perform the deterioration process based on the aperture value.

11. An image processing device according to any one of claims 1 to 10; an image sensor for photoelectrically converting an optical image formed by the optical system,

12. acquiring a captured image and information on shooting conditions including an aperture value of an optical system; obtaining a degradation filter based on the aperture value; performing degradation processing on the captured image based on the degradation filter; When the aperture value is a first aperture value, the degradation amount of the captured image caused by the degradation filter is a first degradation amount, an image processing method characterized in that, when the aperture value is a second aperture value greater than the first aperture value, the amount of degradation of the captured image caused by the degradation filter is a second amount of degradation greater than the first amount of degradation.

13. A program for causing a computer to execute the image processing method according to claim 12.

14. A computer-readable storage medium storing the program according to claim 13.

Images