Image processing device, image processing method, and program

Abstract

To address a problem that, regarding a frequency characteristic caused by an imaging device, in a case where a frequency having no contrast value variation in a section the defocus amount of which is to be calculated is adopted as a processing frequency, the defocus amount cannot be uniquely calculated.SOLUTION: A frequency at which variation of contrast values corresponding to different defocus amounts is larger than a prescribed threshold is determined as a processing frequency. Accordingly, the accuracy of calculating a defocus amount can be improved.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program for recording an image on a recording medium. [Background technology]

[0002] There is known an image processing technique that uses the defocus amount of an image captured by an imaging device. The defocus amount is the difference between the imaging plane and the defocused imaging plane position, i.e., the difference between the planned imaging plane and the actual imaging plane, and is a physical quantity proportional to the amount of blurring of the image.

[0003] Patent Document 1 discloses a technique for performing a three-dimensional correction process on an image output by a printing device by changing a parameter that controls sharpness according to the defocus amount of an input image. In order to use such a technique, it is necessary to correctly obtain the defocus amount from a single image. It is known that the sharpness of edges in an image is high in a focus area (in-focus area) and low in a defocus area (out-of-focus area). Since the sharpness of edges in an image varies depending on not only the defocus amount but also the subject, the defocus amount cannot necessarily be uniquely determined from the sharpness of edges in an image, but it is often used as an effective feature when calculating the defocus amount. For example, Non-Patent Document 1 discloses a technique for calculating a feature representing edge sharpness from the amount of change when blurring an edge detected from an image, and calculating the defocus amount based on the obtained feature. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-146149 A [Non-patent literature]

[0005] [Non-Patent Document 1] Chen,DJ,et al. “FAST DEFOCUS MAP ESTIMATION.”2016 IEEE International Conference on Image Processing(ICIP).IEEE,2016. Summary of the Invention [Problem to be solved by the invention]

[0006] On the other hand, the technique disclosed in Non-Patent Document 1 does not take into account the frequency components for edge detection. Here, the relationship between the sharpness of edges in an image and spatial frequency will be described. FIG. 4 is a graph called MTF (Modulation Transfer Function) that indicates the spatial frequency characteristics that affect the sharpness of image data captured by an imaging device. The horizontal axis indicates spatial frequency, and the vertical axis indicates contrast values ​​that indicate how faithfully the contrast of a subject can be reproduced. As shown in FIG. 4, in an image captured by an imaging device, the contrast value depends on the spatial frequency. That is, in the calculation process of the defocus amount, if a frequency component with a small difference in contrast value in the section for calculating the defocus amount is used, the difference in sharpness of the detected edges becomes small, and the calculation accuracy of the defocus amount decreases.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to improve the accuracy of calculating the defocus amount by selecting frequency components having a large difference in contrast value in the section of the defocus amount to be calculated. [Means for solving the problem]

[0008] The present invention comprises an acquisition means for acquiring image data obtained by imaging using an imaging device, spatial frequency characteristics attributable to the imaging device which indicate the correspondence between spatial frequency and contrast value, and section information which indicates a section for calculating a defocus amount, a determination means for determining a processing frequency based on the spatial frequency characteristics and the section information, a feature calculation means for calculating a feature indicating edge sharpness of a frequency component equivalent to the determined processing frequency based on the image data, and a defocus amount calculation means for calculating a defocus amount related to the image data based on the section information and the feature amount, wherein the determination means determines as the processing frequency a frequency which satisfies that a difference between a first contrast value corresponding to a first defocus amount included in the section and a second contrast value corresponding to a second defocus amount included in the section and different from the first defocus amount is greater than a predetermined threshold value. Effect of the Invention

[0009] The present invention improves the accuracy of defocus amount calculation by performing defocus amount calculation processing using frequency components in which the difference in contrast value is sufficiently large in the section in which the defocus amount is calculated. [Brief description of the drawings]

[0010] [Figure 1] A diagram showing the relationship between the lens and the amount of defocus [Diagram 2] A defocus map and examples of each region [Diagram 3] Block diagram showing the configuration of an image processing system [Figure 4] FIG. 13 is a diagram illustrating an example of spatial frequency characteristics that affect sharpness due to an imaging device in in-focus and out-of-focus areas of an image at a specific defocus amount. [Diagram 5] Image processing flow chart [Figure 6] FIG. 4 is an explanatory diagram of a processing frequency used in the defocus amount calculation process; [Figure 7] A diagram showing an example of a LUT that converts a feature value representing sharpness into a defocus amount. [Figure 8] FIG. 13 is a diagram showing the relationship between the defocus amount and the control parameter for the sharpening strength. [Figure 9] A diagram illustrating the spatial frequency characteristics that affect the sharpness due to the output device in the in-focus and out-of-focus areas of an image at a specific defocus amount. [Figure 10] Image processing flow chart [Figure 11] FIG. 1 is a diagram for explaining a method for setting a defocus amount calculation section based on spatial frequency characteristics that affect sharpness due to an output device. [Figure 12] Block diagram showing the configuration of the control unit [Figure 13] Image processing flow chart [Figure 14] FIG. 13 is a schematic diagram for selecting a processing frequency in each of the divided defocus amount calculation sections. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (First embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] In this embodiment, an inkjet printer is used as an example of an output device. The image processing device according to this embodiment detects edges in an image captured by an imaging device, and calculates a defocus amount (to be described later) based on the sharpness of the image, and generates a defocus map in which the defocus amount is mapped at multiple locations on the input image data. Next, a sharpening process based on a sharpening intensity control parameter is executed on corresponding pixels of the image while referring to the defocus map, and the obtained processed image is output to an output device (inkjet printer). At this time, a frequency with a sufficiently large difference in contrast value on the spatial frequency characteristic that affects the sharpness caused by the imaging device (to be described later) is used as a processing frequency in the section in which the defocus map is calculated, thereby making it possible to suppress a decrease in the calculation accuracy of the defocus amount.

[0013] (Defocus amount and defocus map) The defocus amount and the defocus map will be described with reference to Fig. 1 and Fig. 2. In Fig. 1, a focal plane 101 is a plane that is parallel to an image plane (imaging plane) 103 in an imaging device and is in focus. The defocus amount 105 is the difference between the imaging plane 103 and a defocused imaging plane position 104, that is, the difference between a planned imaging plane and an actual imaging plane, and is proportional to the amount of blurring of the image.

[0014] Fig. 2(a) is a defocus map for input image data obtained by photographing two cubes (201, 202). The above-mentioned defocus amount is mapped at multiple points on the input image data, and information on the defocus amount corresponding to each pixel of the input image data is stored. Fig. 2(b) is a diagram showing the defocus amount and each region.

[0015] In FIG. 2(a), the blackest area 203 is the surface (focused surface) that is in focus at the time of shooting, and the defocus amount is 0. In FIG. 2(b), the area in focus is defined as the in-focus area 203. The color of the image becomes whiter as it moves away from the in-focus surface, and the defocus amount changes. Areas other than the in-focus area 203 are defined as non-focused areas 204, which are areas on the defocus map that do not correspond to the in-focus surface. Furthermore, an area that is allowed to be in focus is defined as an allowable in-focus area 205. The allowable in-focus area 205 may be defined as a depth of field, or may be arbitrarily defined by subject experiments. In this embodiment, areas other than the allowable in-focus area 205 are defined as non-allowable in-focus areas 206.

[0016] (System Configuration) 3 is a diagram showing the configuration of an image processing system to which the image processing device of this embodiment is applied. As shown in this figure, an image processing device 30 of this embodiment is connected to an imaging device 31 and an output device 32.

[0017] The imaging device 31 is an image input device such as a digital camera, which obtains image data of a two-dimensional image including a target object, and outputs the image data to the image processing device 30. For example, the imaging device 31 is configured with an imaging sensor that captures an image of a subject, a memory that stores the captured image data, and the like.

[0018] The image processing device 30 acquires data from the imaging device 31, and performs control instructions to the output device 32, transfer of necessary information and data, etc. The storage unit 305 stores and manages the OS, the system program of this embodiment, various application software, and parameter data required for various processes. This storage unit 305 can be configured with a means represented by a hard disk or a flash ROM. The control unit 302 is configured with a CPU (Central Processing Unit) and the like, and controls the processing in the image processing device 30 by expanding the software stored in the storage unit 305 into the working memory 304 and executing it. The functional configuration of the control unit 302 will be described later with reference to FIG. 3(b). The operation unit 303 (hereinafter also referred to as "UI"), which serves as a user interface, processes input by the user and display to the user with respect to the execution of the above-mentioned processing, and includes input devices such as a keyboard and a mouse and display devices such as a display. The data input / output unit 301 exchanges data with the imaging device 31 and the output device 32. This data input / output unit 301 may input / output data to / from an external recording medium such as an SD card without being connected to the imaging device 31 or the output device 32 .

[0019] The output device 32 is, for example, an inkjet printer, and is composed of a data transfer unit, a printer control unit, a printing unit, etc. The output device 32 prints the image acquired from the image processing device 30 on a medium such as paper by an inkjet recording method. In this embodiment, the case where the output device 32 is an inkjet printer will be described, but various output devices such as a display and a projector can be used.

[0020] FIG. 3(b) is a configuration diagram showing the control unit 302 in this embodiment. The control unit 302 is composed of the following components. The image acquisition unit 306 acquires an input image captured by the imaging device 31 and input to the data input / output unit 301 of the image processing device 30. The characteristic information acquisition unit 307 acquires spatial frequency characteristics affecting sharpness attributable to the imaging device from the storage unit 305 of the image processing device 30. Furthermore, the characteristic information acquisition unit 307 acquires section information indicating a section for calculating the defocus amount. The section information may be information input by the user, or may be information previously stored in the storage unit 305.

[0021] The processing frequency selection unit 308 selects a processing frequency. Here, a frequency is selected in which the difference in contrast value is sufficiently large in the spatial frequency characteristics that affect the sharpness caused by the imaging device in the section in which the defocus amount is calculated. The feature calculation unit 309 detects an edge of a frequency component corresponding to the processing frequency acquired from the processing frequency selection unit 308 from the input image, and calculates a feature that indicates the sharpness of the detected edge.

[0022] The defocus amount calculation unit 310 generates a defocus map by calculating defocus amounts at multiple points on the input image based on the feature amount representing the sharpness of the edge. The image processing unit 311 performs a sharpening process on the corresponding pixel of the input image by using a parameter according to the defocus amount while referring to the defocus amount of the defocus map.

[0023] Note that some or all of the functions of the components of the control unit 302 may be realized by using a dedicated circuit. Also, some of the functions of the components of the control unit 302 may be realized by using a cloud computer.

[0024] (Spatial frequency characteristics that affect sharpness due to imaging devices) 4 is a graph showing spatial frequency characteristics that affect the sharpness of image data captured by the imaging device 31. The spatial frequency characteristics that affect the sharpness refer to characteristics with spatial frequency on the horizontal axis and contrast value on the vertical axis. The contrast value is a value that indicates how faithfully the contrast of a subject can be reproduced. Examples of such characteristics include MTF (Modulation Transfer Function) and CTF (Contrast Transfer Function). The above examples are not limiting as long as it is an index that indicates how faithfully the contrast can be reproduced.

[0025] FIG. 4(a) illustrates the characteristics of the in-focus region where the defocus amount is 0, and FIG. 4(b) illustrates the characteristics of the out-of-focus region at a specific defocus amount. These characteristics change due to degradation of image quality in the imaging optical system, settings due to the imaging device, image processing in the imaging device, etc. Therefore, even with the same defocus amount, the spatial frequency characteristics differ depending on the above conditions. In addition, these spatial frequency characteristics are calculated by a known method as known from the MTF due to the imaging device, and detailed description is omitted.

[0026] The deterioration of image quality in an imaging optical system is due to the OTF (Optical Transfer Function), which includes lenses and various optical filters, in image data obtained by receiving light at an imaging element through the imaging optical system. This imaging optical system may be configured using not only lenses but also mirrors (reflective surfaces) having curvature. The above-mentioned OTF changes depending on the imaging conditions such as the aperture value and focal length of the imaging optical system, and also changes depending on the position of the image and the focus state of the imaging optical system relative to the subject. In this way, the deterioration of image quality in the imaging optical system may change the above-mentioned spatial frequency characteristics.

[0027] In this embodiment, the MTF of the central image height of the imaging optical system will be described as an example of the spatial frequency characteristic that affects the sharpness caused by the imaging device. In reality, since the MTF differs for each position of the image during shooting, including the degradation of image quality in the imaging optical system, it is desirable to obtain the MTF for each position of the image. However, if there is no effect on the processing result between using the MTF for each position of the image and using a part of the image (for example, the central image height of the imaging optical system), the entire image may be processed based on the MTF of a part of the image. In addition, a target MTF may be set without reflecting minute factors that affect the MTF and do not affect the processing result.

[0028] (Explanation of the processing flow) 5 is a flowchart of the overall processing procedure performed by the control unit 302. In this embodiment, an example is shown in which the control unit 302 is located in the image processing device 30, but it may be located in the imaging device 31 or the output device 32. This processing is performed by the control unit 302 reading out and executing a program stored in the storage unit 305.

[0029] (Image Acquisition) First, in step S500, the image acquisition unit 306 acquires an input image input to the image processing device 30. The acquired input image is output to the feature amount calculation unit 309 and the image processing unit 311.

[0030] (Obtaining spatial frequency characteristics) In step S501, the characteristic information acquisition unit 307 acquires the MTF of the imaging optical system as a spatial frequency characteristic affecting sharpness caused by the imaging device from the storage unit 305. Furthermore, it acquires section information indicating a section for calculating the defocus amount. Then, it outputs the acquired MTF and section information to the processing frequency selection unit 308. Note that, although an example in which the MTF is acquired from the storage unit 305 has been shown, if the imaging device 31 stores its own MTF, the MTF may be acquired from the imaging device 31 via the data input / output unit 301.

[0031] (Select the processing frequency) In step S502, the processing frequency selection unit 308 selects a processing frequency. The processing frequency selected here is a frequency at which the difference in contrast value on the MTF acquired in step S501 is sufficiently large for each defocus amount in the section for which the defocus amount is calculated. The selected processing frequency is then output to the feature calculation unit 309.

[0032] Fig. 6 is a diagram for explaining the processing frequency selected by the processing frequency selection unit 308. Fig. 6(a) and (b) show contrast values ​​for each defocus amount at different frequencies. In this embodiment, it is assumed that [d0, dx] (d0, d1, ..., dx-1, dx) is set as the interval for calculating the defocus amount.

[0033] Fig. 6(a) shows contrast values ​​c0, c1, ..., cx-1, cx at frequency fa for defocus amounts d0, d1, ..., dx-1, dx, respectively. Similarly, Fig. 6(b) shows contrast values ​​c0', c1', ..., cx-1', cx' at frequency fb for defocus amounts d0, d1, ..., dx-1, dx, respectively.

[0034] At frequency fa, the contrast values ​​in the range of the calculated defocus amounts satisfy the relationship c0>c1>...>cx-1>cx. In other words, the difference in contrast values ​​on the MTF corresponding to two consecutive defocus amounts is sufficiently large for defocus amounts d0, d1,..., dx-1, dx.

[0035] On the other hand, at frequency fb, the contrast value cx-1' corresponding to the defocus amount dx-1 and the contrast value cx' corresponding to the defocus amount dx are cx-1'=cx'. That is, there is no difference in the contrast values ​​on the MTF corresponding to two different defocus amounts.

[0036] Here, a case where the processing frequency selection unit 308 selects the frequency fb as the processing frequency for calculating the defocus amount will be described. As can be seen from FIG. 6(b), even in areas where the defocus amount at the time of shooting is different, there is no difference in the contrast value at the frequency fb. Therefore, there is no difference in the sharpness of the edge detected at the frequency component corresponding to the frequency fb, and the defocus amount cannot be calculated uniquely. On the other hand, as shown in FIG. 6(a), at the frequency fa, it is possible to detect the difference between two contrast values ​​corresponding to two different defocus amounts, and it is possible to detect the difference in sharpness. Therefore, the processing frequency selection unit 308 of this embodiment selects, as the processing frequency, a frequency at which a difference in contrast value occurs on the MTF of the imaging device, such as the frequency fa.

[0037] As a method for selecting the processing frequency, for example, a frequency may be detected at which the difference in contrast value on the MTF of two consecutive defocus amounts is greater than a predetermined threshold value. At this time, it is preferable to determine a frequency that satisfies that the difference between two contrast values ​​corresponding to two consecutive defocus amounts is greater than a predetermined threshold value Th for a plurality of predefined defocus amounts in the section in which the defocus amount is calculated. That is, it is preferable that (cn-1)-(cn)>Th when n=0 to x.

[0038] When multiple frequencies are detected, the median of the detected frequencies may be selected as the processing frequency, or the frequency at which the difference c0-cx between the maximum and minimum contrast values ​​of the calculated defocus amounts is maximum may be selected. Alternatively, the difference between the contrast values ​​on the MTF for each defocus amount may be calculated, and the frequency at which the variance of the difference is minimum may be selected as the processing frequency. Note that the method of selecting the processing frequency is not limited to the above method, as long as the frequency at which the difference between the contrast values ​​on the MTF for each defocus amount occurs can be selected as the processing frequency.

[0039] (Calculate feature values) Returning to Fig. 5, in step S503, the feature amount calculation unit 309 detects edges of the frequency components corresponding to the processing frequency selected in step S502, and calculates a feature amount representing sharpness. Then, the feature amount representing the obtained sharpness is output to the defocus amount calculation unit 310.

[0040] The frequency f of an image is determined by how many pairs of black and white pixels, called line pairs, can be resolved per mm, and can be expressed by equation (1).

[0041]

number

[0042] Here, Sraw is the size of the raw image, Ssensor is the sensor size of the imaging device, and P is the pixel pitch in the line pair. For example, in the case of a line pair of two pixels, i.e., a line pair with one black and one white pixel, the pixel pitch P=1, and in the case of a line pair of four pixels, i.e., a line pair with two black and two white pixels, the pixel pitch P=2. By modifying formula (1), the pixel pitch P is expressed as formula (2).

[0043]

number

[0044] That is, to detect the edge of the frequency component corresponding to the processing frequency in the image, it is sufficient to calculate a feature value representing sharpness from the edge corresponding to the pixel pitch where f=processing frequency in formula (2). For example, the input image is reduced so that the reduction ratio=(processing frequency) / (maximum frequency that the image can express), that is, the image size becomes f when P=1 in formula (1). As a result, the frequency component corresponding to the processing frequency is converted to one pixel pitch, so that the edge of the frequency component corresponding to the processing frequency can be detected by using a general edge detection filter such as a Sobel filter or a Prewitt filter.

[0045] In this embodiment, a feature value representing sharpness is calculated based on a method using the variance of the second derivative value disclosed in Non-Patent Document 1 "Pech-Pacheco, Jose Luis, et al. "Diatom autofocusing in brightfield microscopy: a comparative study." Proceedings 15th International Conference on Pattern Recognition. ICPR-2000. Vol. 3. IEEE, 2000." The variance of the second derivative value in a certain region is large when the sharpness of the region is high, and is small when the sharpness of the region is low, so it can be used as a feature value representing sharpness.

[0046] First, a region division process is performed on the reduced image to divide it into at least one or more small regions. The region division process may use a known technique such as SLIC (Simple Linear Iterative Clustering), or may simply divide the image into rectangular shapes; the region division method is not limited. Next, a Laplacian filter is applied to each pixel of the reduced image to calculate a secondary differential value. Then, the variance value of the secondary differential value in the small region is set as the value of the small region, and is set as a feature value representing sharpness. In this embodiment, an example is shown in which the variance value of the secondary differential value is used as a feature value representing sharpness, but the average value or median value of the primary differential value may also be used; the feature value to be used is not limited.

[0047] Alternatively, a band-pass filter that extracts only signals of the processing frequency may be applied to the frequency space obtained by Fourier transforming the image, and then the edges of the processing frequency components may be detected by inverse Fourier transforming the image. In this case, the intensity value of the detected edge may be used as a feature value representing the sharpness.

[0048] (Calculate the defocus amount) Next, in step S504, the defocus amount calculation unit 310 calculates defocus amounts at multiple points on the input image based on the feature amount representing the sharpness calculated in step S503, and generates a defocus map. Then, the obtained defocus map is output to the image processing unit 311.

[0049] As a method of calculating the defocus amount [0, dx] from the feature amount [s0, sx] representing the sharpness, there is a method of creating and applying an LUT (Look Up Table). For example, as shown in FIG. 7, an LUT is created that converts the maximum value sx of the feature amount representing the calculated sharpness to the defocus amount = 0 (in focus) and converts the minimum value s0 to the defocus amount = dx (out of focus). By applying such an LUT, the defocus amount can be calculated from the feature amount representing the sharpness. Here, the maximum value and the minimum value of the feature amount representing the sharpness obtained from the input image are used to create the LUT, but an LUT in which the feature amount representing the sharpness corresponding to the defocus amount 0 (in focus) and the defocus amount dx (out of focus) respectively is set in advance may be used. When the LUT is applied, the value converted to the defocus amount outside the calculation range (e.g., a negative value) can be rounded to the value of the defocus amount calculation range. In addition, the LUT is not limited to the two-point LUT of the defocus amounts 0 and dx, but a larger LUT may be created and applied.

[0050] Furthermore, in the present embodiment, the means for calculating the defocus amount using the LUT has been described, but it is also possible to create and use a relational expression between the feature amount representing the sharpness and the defocus amount without using the LUT.

[0051] The method of calculating the defocus amount from the feature value representing the sharpness of the edge is not limited to the method described in step S503 and step S504. For example, the defocus amount may be calculated by applying the defocus amount calculation method described in the above-mentioned Non-Patent Document 1 to the edge of the processing frequency component. Also, any method other than the above may be used as long as it is a method for calculating the defocus amount based on the sharpness of the edge.

[0052] (Sharpening) Next, in step S505, the image processing unit 311 acquires image processing control parameters from the storage unit 305. Different parameters are set for these image processing control parameters according to the defocus amount. Using the acquired image processing control parameters, image processing is performed on the corresponding pixels of the input image while referring to the defocus map calculated by the defocus amount calculation unit 310. Then, the image obtained by this image processing is output to the output device 32.

[0053] In this embodiment, an example of executing a sharpening process as the above image processing will be described in detail. In the sharpening process, for example, a Laplacian Of Gaussian filter (Equation (3) below) or an unsharp mask is used. Equation (4) shows a conversion formula for luminance information of input image data when using a Laplacian Of Gaussian filter. Out(x,y) is image data after sharpening process at pixel (x,y), σ is a control parameter for the standard deviation of the Gaussian function, and β is a control parameter for the sharpening strength.

[0054]

number

[0055] Out(x,y)=I(x,y)+(I(x,y)-h(x,y))×β …Equation (4) By setting different values ​​for each defocus amount as image processing parameters, a control parameter σ of the standard deviation of this Gaussian function and a control parameter β of the sharpening strength, it becomes possible to perform image processing in which the sharpness is controlled according to the defocus amount of the input image.

[0056] FIG. 8 shows an example of setting the control parameter of the sharpening strength. In FIG. 8, β=β1 at defocus amount d=0 indicates the control amount for the in-focus area of ​​the input image. The defocus amount d1 is the boundary value between the permissible in-focus area and the unpermissible in-focus area shown in FIG. 2(b). The defocus amount d2 is the maximum defocus amount included in the defocus map. As shown in this figure, β is set as the control parameter so that it is maximum in the in-focus area, i.e., defocus amount=0, and monotonically decreases as the defocus amount increases. This makes it possible to increase the difference in sharpness between the in-focus area and the unfocused area by the sharpening process, and to generate an image with improved three-dimensional effect.

[0057] The filter used for the sharpening process is not limited to the Laplacian Of Gaussian filter, but there is also a method of adjusting the strength of a specific filter by the sharpness control amount. The specific filter is, for example, a filter created by obtaining the inverse characteristics of the sharpness reduction information of the output device.

[0058] In the above, an example was shown in which the image processing control parameter has a control parameter for sharpening intensity, and sharpening processing is performed on the input image. The image processing control parameter may have a parameter for controlling contrast, and contrast processing is performed on the input image. Alternatively, a form in which either sharpening processing or contrast processing is selected, or the two processing methods are used in combination, may be used.

[0059] It can also be used to detect the in-focus area of ​​an input image. For example, the defocus amount corresponding to the permissible in-focus area 205 shown in FIG. 2(b) is set as a threshold, and when the defocus amount in the defocus map is greater than the threshold, an image processing control parameter is set so that the output of the corresponding pixel in the input image is set to 0. This allows for in-focus area detection processing to detect image areas where the defocus amount is smaller than the threshold.

[0060] In the present embodiment, an example has been shown in which sharpening processing is performed on an input image, but the present invention is not limited to the above processing as long as image processing is performed using different parameters according to the defocus amount.

[0061] As described above, in this embodiment, a frequency at which the difference in contrast value is sufficiently large on the MTF of the imaging device in the section where the defocus amount is calculated is selected as the processing frequency. Then, the defocus amount is calculated based on the edge sharpness of the selected processing frequency component in the image. With this configuration, it is possible to improve the calculation accuracy when calculating the defocus amount from image data.

[0062] In the above embodiment, the characteristic information acquisition unit 307 acquires the MTF of the image capture device, and the processing frequency selection unit 308 selects, as the processing frequency, a frequency at which the difference in contrast value on the MTF is sufficiently large in the section where the defocus amount is calculated. On the other hand, when the MTF of the image capture device can be acquired in advance, the following method may be used.

[0063] The process of acquiring spatial frequency characteristics and section information in step S501 and the process of selecting a processing frequency in step S502 are not performed, and the processing frequency is calculated in advance. Then, the process of calculating a feature amount representing edge sharpness in step S503 is performed on the image data. Here, the feature amount calculation unit 309 acquires the processing frequency information and section information stored in advance in the storage unit 305, and calculates the feature amount. This makes it possible to shorten the processing time.

[0064] In the above embodiment, the image processing unit 311 refers to the calculated defocus map and performs processing based on different image processing parameters according to the defocus amount for the corresponding pixels of the image, but image processing does not necessarily have to be performed. When it is desired to obtain the defocus amount of an image, it is sufficient to obtain the defocus map, and it is not necessary to perform the sharpening process in step S505. This can shorten the processing time.

[0065] Second embodiment The image processing device according to this embodiment executes image processing for correcting a decrease in sharpness caused by outputting an image captured by an imaging device using an output device. Here, an example of executing image processing for correcting a decrease in sharpness of a printed matter (hereinafter referred to as "sharpness of a printed matter") caused when printing using an inkjet printer will be described.

[0066] First, the spatial frequency characteristics that affect the sharpness of the print are obtained. Next, based on the obtained spatial frequency characteristics, the defocus amount that causes a large decrease in the sharpness of the print is specified, and this section is set as the section for calculating the defocus amount. Hereinafter, the processing of this embodiment will be described with reference to Figs. 9 to 11.

[0067] The configuration of the image processing device is the same as that of the first embodiment, but the characteristics information acquisition unit 307 and the processing frequency selection unit 308 in the control unit 302 have different functions from those of the first embodiment, and therefore their functions will be described below. The other configurations have the same functions as those of the first embodiment, and therefore their description will be omitted. Only the differences from FIG. 3(b) will be described below.

[0068] The characteristic information acquisition unit 307 acquires the spatial frequency characteristics affecting the sharpness caused by the imaging device and the spatial frequency characteristics affecting the sharpness caused by the output device from the storage unit 305 in the image processing device 30 shown in Fig. 3(a). The processing frequency selection unit 308 first identifies a section of the defocus amount in which the sharpness reduction caused by the output device is large based on the spatial frequency characteristics affecting the sharpness caused by the output device. Then, the specified section is set as a section for calculating the defocus amount. Next, in the acquired section for calculating the defocus amount, a frequency at which a difference occurs in the contrast value on the spatial frequency characteristics affecting the sharpness caused by the imaging device is selected as a processing frequency.

[0069] The spatial frequency characteristics that affect the sharpness caused by the output device are indices that indicate how faithfully the contrast can be reproduced, similar to the spatial frequency characteristics that affect the sharpness caused by the imaging device described above. When an image is output through an output device such as a printer, the sharpness of the image is reduced due to output characteristics such as the recording medium and ink bleeding, and due to a resolution conversion process that resizes the input image data to the recording medium size (print size). A similar reduction in sharpness occurs in the output image with a display or projector.

[0070] Fig. 9 is a diagram showing spatial frequencies that affect sharpness caused by an output device according to the defocus amount of an input image. Fig. 9(a) illustrates the spatial frequency characteristics that affect the sharpness caused by the output device in a focused region (defocus amount = 0), and Fig. 9(b) illustrates the spatial frequency characteristics that affect the sharpness caused by the output device at a specific defocus amount in a non-focused region.

[0071] The change in sharpness of the input image based on the spatial frequency characteristics that affect the sharpness caused by the output device changes significantly in areas where the input data has high sharpness, i.e., in the permissible in-focus area where the defocus amount is small. On the other hand, there is almost no change in areas where the input data has low sharpness, i.e., in the unpermissible in-focus area where the defocus amount is large. Therefore, as shown in Figure 9, the spatial frequency that affects the sharpness caused by the output device shows different characteristics depending on the defocus amount of the input image.

[0072] In this embodiment, the MTF of an inkjet printer will be taken as an example of the spatial frequency characteristic that affects the sharpness due to the output device.

[0073] Fig. 10 is a flowchart of the overall processing procedure performed by the control unit 302 in this embodiment. Note that steps S1000, S1001, and S1004 to S1006 are similar to steps S500, S501, and S502 to S504 in the first embodiment, respectively, and therefore their explanations are omitted. Only the differences from the flowchart in Fig. 5 will be explained below.

[0074] In step S1002, characteristic information acquisition unit 307 acquires the MTF of the inkjet printer as the spatial frequency characteristic affecting the sharpness attributable to the output device from storage unit 305. Then, the acquired MTF is output to processing frequency selection unit 308. In the present embodiment, an example has been shown in which the MTF is acquired from storage unit 305, but if output device 32 stores its own MTF, the MTF may be acquired from output device 32 via data input / output unit 301.

[0075] In step S1003, the processing frequency selection unit 308 identifies a defocus amount section in which the sharpness reduction caused by the output device is large based on the MTF of the output device acquired in step S1002. Then, the identified defocus amount section is set as a section for calculating the defocus amount in this embodiment. Next, a frequency in which the difference in contrast value is large on the MTF of the imaging device acquired in step S1001 in the acquired section for calculating the defocus amount is selected as a processing frequency. Then, the selected processing frequency is output to the feature calculation unit 309.

[0076] 11 is a diagram showing the MTF of an output device. An example of a method for identifying a defocus amount section in which the sharpness degradation caused by the output device is large, based on the MTF of the output device acquired in step S1002, will be described with reference to this diagram.

[0077] First, a reference frequency fp is set. For example, 300 dpi or 600 dpi is conceivable, but the frequency fp is not limited to this. Contrast values ​​corresponding to the defocus amounts d0 (focused), d1, d2, and d3 (unfocused) at the frequency fp are indicated as c0, c1, c2, and c3, respectively. Next, these contrast values ​​are compared with a threshold ct, which is a contrast value that allows a decrease in sharpness that has been set in advance. In a defocus amount with a contrast value smaller than the threshold ct, it is determined that the decrease in sharpness caused by the output device is large, and is set as the section for calculating the defocus amount in this embodiment. In FIG. 11, c0 to c2 indicate values ​​smaller than the threshold ct, so the defocus amount calculation section can be set as [d0, d2].

[0078] The above-mentioned method of setting the defocus amount calculation section is an example, and the defocus amount calculation section may be set by other methods. For example, an image corresponding to the amount of blur corresponding to each defocus amount in FIG. 11 may be created, and an output by an actual output device may be compared with an input image, and a section where sharpness is felt to be significantly reduced may be set as the defocus amount calculation section. In addition, the above-mentioned method of setting the defocus amount calculation section is not limited as long as the calculation section is set to a defocus amount determined based on the spatial frequency characteristics that affect the sharpness caused by the output device.

[0079] The method of selecting a frequency at which a difference in contrast value occurs on the MTF in the section in which the acquired defocus amount is calculated is the same as in the first embodiment described above, and therefore a description thereof will be omitted.

[0080] In step S1007, the image processing unit 311 acquires a sharpening intensity control parameter as an image processing parameter from the storage unit 305. Then, using the acquired sharpening intensity control parameter, and referring to the defocus map calculated by the defocus amount calculation unit 310, executes sharpening processing on the corresponding pixel of the input image. Then, the sharpened image obtained by the processing is output to the output device 32.

[0081] Since the processing other than the method of setting the image processing parameters is the same as step S505 in the first embodiment, the description will be omitted and only the differences from step S505 will be described.

[0082] In this embodiment, a parameter for restoring the sharpness of the input image based on the MTF of the output device is set as the control parameter of the above-mentioned sharpening strength. For example, a control parameter of the sharpening strength for restoring the sharpness of the focused area of ​​the output image at a specific frequency, the frequency characteristic to the sharpness of the input image, or the frequency characteristic by sharpening processing is set from the MTF of the imaging device and the output device. Similarly, for the non-focused area, the restoration amount is calculated from the MTF of the imaging device and the output device obtained for each defocus amount, and is set as the control parameter of the sharpening strength. In this way, image processing is performed to correct the decrease in sharpness caused by the output device.

[0083] As described above, in this embodiment, the spatial frequency characteristics that affect the sharpness caused by the inkjet printer, which is the output device, are obtained, and a section where the sharpness loss caused by the inkjet printer is large is set as a defocus amount calculation section based on the spatial frequency characteristics. This makes it possible to calculate a defocus amount appropriate for correcting the sharpness loss caused by the inkjet printer. Using the calculated defocus amount and a control parameter for the sharpening strength based on the MTF of the inkjet printer, the sharpness loss caused by the inkjet printer can be appropriately corrected, and the sharpness of the printout can be improved.

[0084] In the above embodiment, the characteristic information acquisition unit 307 acquires the MTF of the output device, and the processing frequency selection unit 308 sets the section for calculating the defocus amount based on the MTF of the output device. However, this is not necessarily limited to the above, if the MTF of the output device can be acquired in advance. That is, the section for calculating the defocus amount may be calculated in advance without performing the acquisition process of the spatial frequency characteristic affecting the sharpness due to the output device in step S1001 and the setting process of the section for calculating the defocus amount in step S1002. At this time, in the selection process of the processing frequency in step S1004, the feature amount calculation unit 309 acquires the section information for calculating the defocus amount stored in advance in the storage unit 305 and performs the process. This can shorten the processing time.

[0085] In addition, in this embodiment, an inkjet printer is taken as an example of an output device, and a system for correcting a decrease in sharpness is described, but the output device is not limited to an inkjet printer and may be a display or a projector. Similarly, in a display or projector, image sharpness may decrease due to resolution conversion processing or the like that changes the size to the output medium. By applying this system, it is possible to appropriately correct the sharpness in the output device, as in the case of an inkjet printer.

[0086] (Third embodiment) The image processing device according to this embodiment acquires in-focus area information of an input image, and performs processing so that the defocus amount in the acquired in-focus area becomes the minimum value (=0). Fig. 12 is a block diagram of a control unit 302 of the image processing device according to this embodiment. The functions of an image acquisition unit 306, a characteristic information acquisition unit 307, a processing frequency selection unit 308, a feature amount calculation unit 309, a defocus amount calculation unit 310, and an image processing unit 311 in this embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted. Only the differences from Fig. 3(a) will be described below.

[0087] Fig. 13 is a flowchart of the overall processing procedure performed by the control unit 302 in this embodiment. Note that steps S1300 to S1303 and step S1306 are similar to steps S500 to S503 and step S505 in the first embodiment, respectively, and therefore their explanations are omitted. Only the differences from the flowchart in Fig. 5 will be explained below.

[0088] In step S1304, the in-focus region acquisition unit 1201 acquires in-focus region information indicating a portion of the input image that was in focus at the time of shooting from the incidental information at the time of shooting. Then, the in-focus region acquisition unit 1201 outputs the acquired in-focus region information to the defocus amount calculation unit 310.

[0089] In this embodiment, an example is shown in which focused area information is obtained from EXIF ​​(Exchangeable Image File Format), which is an image file format that includes metadata at the time of shooting. Focused area information refers to information that can identify the focused area in an image, such as the start point of a rectangular area and its height and width. In EXIF, there is a tag that records the position information of the ranging point used for AF (Auto Focus) at the time of shooting. It is possible to obtain focused area information by referring to the tag that records the position information of the ranging point that was in focus at the time of shooting.

[0090] If the in-focus area information cannot be obtained by referring to only one tag, the in-focus area information may be obtained by referring to multiple tags in the EXIF. Also, if the in-focus area is divided into multiple areas, the in-focus area information may be obtained for all areas or for only one area.

[0091] In addition, the method of acquiring the in-focus area information is not limited to the method of acquiring the in-focus area information from EXIF, which is incidental information at the time of shooting. For example, the user may specify the in-focus area information in the input image using UI 303, or the in-focus area information may be estimated by analyzing the image.

[0092] In step S1305, the defocus amount calculation unit 310 calculates defocus amounts at multiple points on the input image based on the feature amount representing the sharpness calculated in step S1303, and generates a defocus map. At this time, the defocus amount is calculated using the in-focus area information acquired in step S1304 so that the feature amount value in the in-focus area in the input image is converted to the minimum value of the defocus amount (=0). Then, the obtained defocus map is output to the image processing unit 311.

[0093] When creating an LUT that converts the feature amount representing sharpness calculated in step S1303 into a defocus amount, the in-focus area information acquired in step S1304 is used. First, a representative value (e.g., average or median) of the feature amount representing sharpness in the in-focus area of ​​the image is calculated. Next, an LUT is created that converts the obtained representative value into the minimum defocus amount value (=0). Note that when there are multiple pieces of in-focus area information, the representative value of the feature amount representing sharpness may be calculated using all the in-focus areas, or a representative value in one area may be calculated.

[0094] The processing other than the method of creating the LUT corresponding to the minimum value (=0) of the defocus amount is the same as in the first embodiment, and therefore the description thereof will be omitted here.

[0095] As described above, by acquiring the in-focus area in the image, it is possible to correctly calculate the defocus amount in the in-focus area as 0. In other words, even in an image with a large amount of noise where the feature amount representing sharpness is calculated to be high outside the in-focus area, the defocus amount can be calculated with high accuracy. As a result, the accuracy of image processing performed using the calculated defocus amount is improved.

[0096] (Other embodiments) In the above embodiment, the defocus map for the calculation section is created by dividing the section in which the defocus amount is calculated, calculating a defocus map for each divided section, and synthesizing the obtained defocus maps for each divided section. For example, as shown in FIG. 14, the frequency f1 has a large difference in contrast value between the defocus amounts d0 to d2, and the resolution between the defocus amounts is high. However, the contrast values ​​are equal between d2 to d4, and the defocus amounts cannot be distinguished. On the other hand, the frequency f2 has a small difference in contrast value between the defocus amounts d0 to d2, and the resolution between the defocus amounts is low, but the difference in contrast value between the defocus amounts d2 to d4 is sufficiently large, and the resolution between the defocus amounts is high. Here, the first defocus map is calculated using f1 as the first processing frequency, and the second defocus map is calculated using f2 as the second processing frequency. Then, the defocus map for the section in which the defocus amount is calculated can be created by synthesizing the areas d0 to d2 in the first defocus map and d2 to d4 in the second defocus map. Although an example has been given in which the section for calculating the defocus amount is divided into two, the number of divisions is not limited to this.

[0097] As described above, by dividing the section for calculating the defocus amount and synthesizing the defocus maps calculated for each divided section, it is possible to improve the calculation accuracy of the defocus amount. [Explanation of symbols]

[0098] 31 Imaging device 31 Image processing device 32 Output Device

Claims

1. an acquisition means for acquiring image data obtained by capturing an image using an imaging device, spatial frequency characteristics indicating a correspondence between spatial frequency and contrast value and attributable to the imaging device, and section information indicating a section for calculating a defocus amount; a determination means for determining a processing frequency based on the spatial frequency characteristic and the section information; a feature value calculation means for calculating a feature value representing edge sharpness of a frequency component corresponding to the determined processing frequency based on the image data; a defocus amount calculation means for calculating a defocus amount for the image data based on the section information and the feature amount; Equipped with The image processing device characterized in that the determination means determines, as the processing frequency, a frequency that satisfies the requirement that a difference between a first contrast value corresponding to a first defocus amount included in the section and a second contrast value corresponding to a second defocus amount included in the section and different from the first defocus amount is greater than a predetermined threshold.

2. 2. The image processing device according to claim 1, wherein the determining means determines, as the processing frequency, a frequency that satisfies the condition that a difference between two contrast values ​​corresponding to two consecutive defocus amounts, respectively, among a plurality of defocus amounts included in the section, is greater than the predetermined threshold value.

3. 3. The image processing device according to claim 1, wherein the feature calculation means calculates a second derivative value of an edge of a frequency component corresponding to the processing frequency from the image data, divides the image data into at least one small region, and calculates a variance value of the second derivative value of the edge contained in each of the small regions as the feature value.

4. 4. The image processing apparatus according to claim 1, further comprising an image processing unit that executes image processing on the image data based on the calculated defocus amount.

5. 5. The image processing apparatus according to claim 4, wherein said image processing means executes image processing in which sharpness monotonically decreases as the defocus amount increases.

6. 5. The image processing apparatus according to claim 4, wherein said image processing means executes image processing in which contrast monotonically decreases as the defocus amount increases.

7. 5. The image processing apparatus according to claim 4, wherein the image processing means executes image processing for detecting an image area in which a defocus amount is smaller than a predetermined value.

8. 8. The image processing apparatus according to claim 1, wherein the acquisition means acquires the section information based on spatial frequency characteristics that affect sharpness due to an output device.

9. 9. The image processing apparatus according to claim 8, wherein the spatial frequency characteristic caused by the imaging device and the spatial frequency characteristic caused by the output device and affecting sharpness are MTFs.

10. 9. The image processing apparatus according to claim 8, wherein the spatial frequency characteristic caused by the imaging device and the spatial frequency characteristic caused by the output device and affecting sharpness are CTFs.

11. The image data further includes a focusing area acquisition unit for acquiring focusing area information indicating a focusing area that was focused when the image data was captured.

11. The image processing apparatus according to claim 1, wherein the defocus amount calculation means calculates a defocus amount for the image data so that a defocus amount of the in-focus region becomes a minimum value.

12. 12. The image processing apparatus according to claim 11, wherein the in-focus area acquisition means acquires the in-focus area information from a tag in an EXIF ​​file that holds supplementary information at the time of photography.

13. 12. The image processing apparatus according to claim 11, wherein the in-focus area acquisition means acquires an area designated by a user as the in-focus area.

14. an acquisition means for acquiring image data obtained by capturing an image using an imaging device, spatial frequency characteristics indicating a correspondence between spatial frequency and contrast value and attributable to the imaging device, and section information indicating a section for calculating a defocus amount; a determination means for determining a processing frequency based on the spatial frequency characteristic and the section information; a feature value calculation means for calculating a feature value representing edge sharpness of a frequency component corresponding to the determined processing frequency based on the image data; a defocus amount calculation means for calculating a defocus amount for the image data based on the section information and the feature amount; Equipped with The image processing device characterized in that the determination means determines, as the processing frequency, a frequency that satisfies the condition that a difference between two contrast values ​​corresponding to two consecutive defocus amounts, respectively, among a plurality of defocus amounts included in the section, is greater than a predetermined threshold.

15. 1. An image processing device for calculating a defocus amount for a predetermined section based on image data obtained by capturing an image using an imaging device, a feature value calculation means for calculating a feature value representing edge sharpness of a frequency component corresponding to a predetermined processing frequency based on the image data; a defocus amount calculation means for calculating a defocus amount for the predetermined section based on the feature amount; Equipped with The image processing device is characterized in that the specified processing frequency is a frequency determined based on spatial frequency characteristics attributable to the imaging device, which indicate the correspondence between spatial frequency and contrast value, and section information indicating the specified section, and is a frequency at which a difference between a first contrast value corresponding to a first defocus amount included in the specified section and a second contrast value corresponding to a second defocus amount included in the specified section and different from the first defocus amount is greater than a specified threshold.

16. an acquisition step of acquiring image data obtained by capturing an image using an imaging device, spatial frequency characteristics attributable to the imaging device that indicate a correspondence between spatial frequency and contrast value, and section information that indicates a section for calculating a defocus amount; a determination step of determining a processing frequency based on the spatial frequency characteristic and the section information; a feature value calculation step of calculating a feature value representing edge sharpness of a frequency component corresponding to the determined processing frequency based on the image data; a defocus amount calculation step of calculating a defocus amount related to the image data based on the section information and the feature amount; Equipped with an image processing method, characterized in that in the determination step, a frequency that satisfies the requirement that a difference between a first contrast value corresponding to a first defocus amount included in the section and a second contrast value corresponding to a second defocus amount included in the section and different from the first defocus amount is greater than a predetermined threshold is determined as the processing frequency.

17. 17. The image processing method according to claim 16, wherein in the determination step, a frequency that satisfies the condition that a difference between two contrast values ​​corresponding to two consecutive defocus amounts in a plurality of defocus amounts included in the section is greater than the predetermined threshold is determined as the processing frequency.

18. an acquisition step of acquiring image data obtained by capturing an image using an imaging device, spatial frequency characteristics attributable to the imaging device that indicate a correspondence between spatial frequency and contrast value, and section information that indicates a section for calculating a defocus amount; a determination step of determining a processing frequency based on the spatial frequency characteristic and the section information; a feature value calculation step of calculating a feature value representing edge sharpness of a frequency component corresponding to the determined processing frequency based on the image data; a defocus amount calculation step of calculating a defocus amount related to the image data based on the section information and the feature amount; Equipped with an image processing method, characterized in that in the determination step, a frequency that satisfies the condition that a difference between two contrast values ​​corresponding to two consecutive defocus amounts, respectively, among a plurality of defocus amounts included in the section, is greater than a predetermined threshold is determined as the processing frequency.

19. 1. An image processing method for calculating a defocus amount for a predetermined section based on image data obtained by capturing an image using an imaging device, comprising: a feature amount calculation step of calculating a feature amount representing edge sharpness of a frequency component corresponding to a predetermined processing frequency based on the image data; a defocus amount calculation step of calculating a defocus amount for the predetermined section based on the feature amount; Equipped with The image processing method is characterized in that the predetermined processing frequency is a frequency determined based on spatial frequency characteristics attributable to the imaging device, which indicate the correspondence between spatial frequency and contrast value, and on section information indicating the predetermined section, and is a frequency at which a difference between a first contrast value corresponding to a first defocus amount included in the predetermined section and a second contrast value corresponding to a second defocus amount included in the predetermined section and different from the first defocus amount is greater than a predetermined threshold value.

20. A program for causing a computer to execute each step of the image processing method according to any one of claims 16 to 19.

21. a first acquisition step of acquiring edge sharpness of a frequency component corresponding to a predetermined processing frequency based on image data acquired by capturing an image using an imaging device; a second acquisition step of acquiring a defocus amount corresponding to each pixel of the image data based on the edge sharpness; Equipped with the processing frequency is a frequency determined based on a spatial frequency characteristic of the imaging device that indicates contrast with respect to a spatial frequency for each defocus amount, an image processing method characterized in that, in the spatial frequency characteristics, a difference between a first contrast value corresponding to a first defocus amount of the processing frequency and a second contrast value corresponding to a second defocus amount different from the first defocus amount of the processing frequency is greater than a predetermined threshold value.

22. 22. The image processing method according to claim 21, wherein the defocus amount acquired in the second acquisition step is within a predetermined range.

23. The image processing method according to claim 22, wherein in the spatial frequency characteristics, a difference between two contrast values ​​at the processing frequency corresponding to two consecutive defocus amounts included in the specified range is greater than the specified threshold value.

24. 22. The image processing method according to claim 21, wherein in the first obtaining step, a second derivative value of an edge of a frequency component corresponding to the processing frequency is obtained from the image data, the image data is divided into at least one or more small regions, and a variance value of the second derivative value of the edge contained in each of the small regions is obtained as the edge sharpness.

25. 22. The image processing method according to claim 21, further comprising the step of executing image processing on the image data based on the defocus amount acquired in the second acquisition step.

26. 26. The image processing method according to claim 25, wherein the image processing is a process in which the sharpness monotonically decreases as the defocus amount increases.

27. 26. The image processing method according to claim 25, wherein the image processing is a process in which the contrast monotonically decreases as the defocus amount increases.

28. 26. The image processing method according to claim 25, wherein the image processing is a process for detecting an image area in which a defocus amount is smaller than a predetermined value.

29. In the first acquisition step, section information indicating a section for calculating a defocus amount is further acquired based on the spatial frequency characteristics of the output device, 22. The image processing method according to claim 21, wherein in the second obtaining step, a defocus amount corresponding to each pixel of the image data is obtained further based on the section information.

30. 22. The image processing method according to claim 21, wherein the spatial frequency characteristic of the image pickup device is MTF.

31. 22. The image processing method according to claim 21, wherein the spatial frequency characteristic of the imaging device is a CTF.

32. A focusing area acquisition step of acquiring focusing area information indicating a focusing area focused at the time of capturing the image data, 22. The image processing method according to claim 21, wherein the minimum value of the defocus amount of each pixel of the image data acquired in the second acquisition step is a pixel in the in-focus region.

33. 33. The image processing method according to claim 32, wherein the in-focus area is acquired from a tag in an EXIF ​​that holds supplementary information at the time of shooting.

34. 33. The image processing method according to claim 32, wherein the in-focus area is obtained in response to a user instruction.

35. a first acquisition means for acquiring edge sharpness of a frequency component corresponding to a predetermined processing frequency based on image data acquired by capturing an image using an imaging device; a second acquisition means for acquiring a defocus amount corresponding to each pixel of the image data based on the edge sharpness; Equipped with the processing frequency is a frequency determined based on a spatial frequency characteristic of the imaging device that indicates contrast with respect to a spatial frequency for each defocus amount, An image processing device characterized in that, in the spatial frequency characteristics, a difference between a first contrast value corresponding to a first defocus amount of the processing frequency and a second contrast value corresponding to a second defocus amount different from the first defocus amount of the processing frequency is greater than a predetermined threshold value.

Images