Image processing device, imaging apparatus, image processing method, and program

Abstract

To provide an image processing device that can express a blurred effect appropriately with high resolution.SOLUTION: An image processing device (100) has acquisition means (108) for acquiring distance information relative to a focus position and filter processing means (106) for performing filter processing on image data from an imaging element (102) using an aberration filter determined on the basis of the distance information and optical information of an optical system (101). The aberration filter has parameters that adjust the characteristics of a first blurred image (204) formed on an image plane due to defocusing at a position closer than the focus position and the characteristics of a second blurred image (205) formed on an image plane due to defocusing at a position closer to infinity than the focus position so that they are closer to each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for adding background blur using a filter appropriate for the circuit scale by changing the number of times of filter processing and the filter accuracy according to the subject distance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6516410 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, lens devices are often designed with emphasis on resolution. However, when designing a lens device with emphasis on resolution, it is difficult to match the characteristics of the front blurred image and the characteristics of the back blurred image. Although Document 1 describes background blur, it does not describe how to express an appropriate blur by matching the characteristics of the front blurred image and the characteristics of the back blurred image.

[0005] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to provide an image processing device capable of appropriately expressing blur with high resolution. [Means for solving the problem]

[0006] An image processing device as one aspect of the present invention has an acquisition means for acquiring distance information for a focus position, and a filter processing means for performing filter processing on image data from an imaging element using an aberration filter determined based on the distance information and optical information of an optical system, the aberration filter having parameters for adjusting the characteristics of a first blurred image formed on an image plane due to defocusing on the closer side than the focus position and the characteristics of a second blurred image formed on the image plane due to defocusing on the infinity side than the focus position so as to approximate each other.

[0007] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device capable of appropriately expressing blur with high resolution. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram of an image processing device according to a first embodiment. [Diagram 2] 5A to 5C are diagrams illustrating the relationship between spherical aberration and a blurred image in the first embodiment. [Diagram 3] FIG. 4 is an explanatory diagram of the relationship between defocus and blur in the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram showing a change in MTF at a certain defocus position when an aberration adding filter is applied in the first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram showing a change in MTF at a certain frequency when an aberration-imparting filter in the first embodiment is applied. [Figure 6] 4 is a flowchart of a filter process in the first embodiment. [Figure 7] 4 is a flowchart of a determination process in the first embodiment. [Figure 8] FIG. 11 is a block diagram of an image processing device according to a second embodiment. [Figure 9]FIG. 13 is a block diagram of an image processing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0011] First Embodiment First, an image processing device 100 according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram of the image processing device 100. The image processing device 100 has an imaging lens 101, an imaging element 102, an A / D converter 103, a signal processing unit 104, a memory (storage unit) 105, a DSP (Digital Signal Processor) 106, an output unit 107, and a CPU (acquisition means) 108. In this embodiment, the DSP 106 is a filtering means that performs filtering on image data using an aberration filter determined based on distance information and optical information of the imaging lens 101, as described later.

[0012] The imaging lens 101 is an optical system (imaging optical system) composed of a group of lenses including a zoom lens, a focus lens, a shift lens, an aperture stop, etc. The imaging element 102 is a photoelectric conversion element such as a CCD sensor or a CMOS sensor that converts an optical image (subject image) formed by the imaging lens 1010 into an electrical signal (analog signal), and has a plurality of pixels (pixel array) arranged two-dimensionally. Each pixel of the imaging element 102 has two photoelectric conversion units, and an electrical signal for phase difference detection can be read out from each photoelectric conversion unit. The A / D converter 103 converts the analog signal output from the imaging element 102 into a digital signal.

[0013] The signal processing unit 104 performs signal processing such as resizing processing for performing predetermined pixel interpolation and reduction, and color conversion processing on the data (digital signal) output from the A / D converter 103. The data output from the A / D converter 103 is directly written to the memory 105 via the signal processing unit 104. The DSP 106 performs various types of image processing on the data output from the signal processing unit 104 or data acquired from the signal processing unit 104 via the memory 105. The DSP 106 also stores the data (image data) that has been subjected to the image processing in the memory 105, or outputs it to the output unit 107.

[0014] The memory 105 stores image data or various data processed by the DSP 106. The memory 105 has a sufficient storage capacity for storing various data. The memory 105 also includes a non-volatile memory. The non-volatile memory is an electrically erasable and recordable memory, such as an EEPROM. The non-volatile memory stores constants and programs for the operation of the DSP 106 or the CPU 108. The programs include, for example, programs for causing the CPU 108, which is a computer, to execute the operations of each part of the image processing device 100. The output unit 107 outputs the image data processed by the DSP 106 to an external device.

[0015] The CPU 108 controls each unit of the image processing device 100. The CPU 108 executes each process described below based on a program stored in the memory 105. The memory 105 includes a system memory. A RAM is used as the system memory, and constants and variables for the operation of the CPU 108, programs read from the non-volatile memory, and the like are expanded in the RAM.

[0016] (Relationship between spherical aberration and blurred image) Next, the relationship between spherical aberration and a blurred image will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram of the relationship between spherical aberration and a blurred image, and shows how light rays that have passed through the upper half of the imaging lens 101 form an image on the imaging surface.

[0017] In Fig. 2, 200 denotes the spherical aberration of the imaging lens 101. As the image height increases (moves away in the direction perpendicular to the optical axis), the light rays defocus in response to the spherical aberration 200, causing blurring. When the imaging plane is located at the lens focal position 201, blurring is minimized, and the imaging lens 101 is generally designed to be focused within the permissible circle of confusion. When the imaging plane is located forward or backward from the focal position 201 (positions 202 and 203), the imaging position of the light rays deviates and spreads in the image height direction, resulting in a blurred image (foreground blurred image (first blurred image) 204, background blurred image (second blurred image) 205).

[0018] At this time, the densities of the light rays forming an image at positions 202 and 203 are different, so the foreground blurred image and the background blurred image have different appearances (optical characteristics such as aberration characteristics). More specifically, in the background blur 205, the width of the light rays forming an image changes gradually, resulting in a blurred image with gradation. On the other hand, in the foreground blur 204, light rays 206 and 207 overlap, resulting in a blurred image with a prominent peripheral outline (double-line blur).

[0019] Here, in order to optically match the shape of the front blur and the shape of the rear blur with respect to the focal position of the imaging lens 101, it is necessary to make the spherical aberration zero. When the spherical aberration is zero, the spherical aberration 200 becomes linear in the image height direction, and the appearance of the front and rear blurred images matches. However, it is generally technically difficult to design a lens with zero spherical aberration, and instead there are problems such as the lens becoming larger. Therefore, generally, the spherical aberration 200 does not become linear in the image height direction, and has a characteristic of spreading in front of or behind the focal position 201. At this time, the front blurred image and the rear blurred image look different from each other. That is, when the front blurred image becomes soft, the rear blurred image becomes sharp (two-line blur), and on the other hand, when the front blurred image becomes sharp (two-line blur), the rear blurred image tends to become soft.

[0020] (Relationship between defocus and blur) Next, the relationship between defocus and blur will be described with reference to Figures 3(a) and 3(b). Figures 3(a) and 3(b) are explanatory diagrams of the relationship between defocus and blur, and show the relationship between a subject in real space and an image on an imaging plane.

[0021] The image sensor 102 is disposed on an image pickup surface 301, and light rays emitted from a subject pass through the imaging lens 101 to form an image on the image pickup surface 301. The subject 302 is a subject focused on the image pickup surface 301, the subject 303 is a subject located on the infinity side of the focused subject, and the subject 304 is a subject located on the close side. When the magnitude of the distance from the subject's imaging position to the image pickup surface is |ε|, ε is defined as the defocus amount. The focused subject 302 has ε=0, and is imaged at a point 305 on the image pickup surface. The subject 303 on the infinity side has ε<0 (front focus, back blur), and is imaged with a width of c1 in the image height direction from the point 305, and is imaged with a blur. The subject 304 on the close side has ε>0 (back focus, foreground blur), and is imaged with a width of c2 in the image height direction from the point 305, and is imaged with a blur.

[0022] Considering the subject 303 at infinity, from the lens formula (mapping formula), 1 / ds+1 / dt=1 / f where f is the focal length of the imaging lens 101. Also, the light rays passing through the lens from the subject 303 intersect with the optical axis on the imaging surface side, and form an image on the imaging surface. At this time, the two triangles formed by the light rays on the imaging surface side from the lens are similar to each other, so if the subject distance to the focused subject 302 is ds, the image distance is dt, the defocus amount is ε1, and the aperture diameter of the imaging lens 101 is D, then (dt-ε1):ε1=D:c1 Here, the image distance dt is the distance from the imaging surface 301 to the imaging lens 101, and the subject distance ds is the distance from the subject 302 to the imaging lens 101. Therefore, if the shooting distance of the imaging lens 101 is L, then the spread of blur c1 is expressed as follows, from the shooting distance L=ds+dt and the aperture value Fno=f / D: c1=(ε1 / (dt-ε1))×(f / Fno), ε1<0 dt=(L±√(L^2-4fL)) / 2 This is expressed as:

[0023] Similarly, if the defocus amount is ε2, the spread of the blur c2 is c2=(ε2 / (dt+ε2))×(f / Fno), ε2>0 Therefore, the width of the blur spread is determined by the defocus amount, focal length, shooting distance, and aperture value. It is considered that the aberration characteristic data has characteristics related to the defocus amount, focal length, shooting distance, and aperture value. It is also possible to design a filter to bring the blur closer in areas where the distance from the in-focus position (ε1 and ε2) is close, or to bring c1 and c2 themselves closer.

[0024] (Relationship between blur and filter processing) Next, the relationship between blur and filter processing will be described. If the actual subject is R, the degradation characteristic (optical transfer function) due to blur is F, and the image data from the image sensor 102 is P, the relationship between them is expressed by a convolution filter of the degradation characteristic F, and in the frequency domain after Fourier transformation, it is expressed as a product as follows: P=F(R) It is expressed as:

[0025] If the subject R is an ideal point light source, the blur characteristics of the subject 303 on the infinitely far side are F1 and the image data are P1, and the blur characteristics of the subject 304 on the close side are F2 and the image data are P2, the degraded images due to blur are respectively as follows: P1=F1(R) P2=F2(R) The aberration imparting filter F21 that converts a front blurred image into a rear blurred image is as follows: F12=P1(P2-1)=F1(R)(P2-1)=F1(F2-1)(P2)(P2-1) =F1(F2-1) Similarly, the aberration imparting filter F12 that converts a back blurred image to a front blurred image is calculated as follows: F21=P2(P1-1)=F2(R)(P1-1)=F2(F1-1)(P1)(P1-1) =F2(F1-1) This allows for mutual conversion between a front blurred image and a rear blurred image.

[0026] (Relationship between MTF and defocus) Next, the relationship between MTF and defocus will be described with reference to Fig. 4(a), (b) and Fig. 5(a), (b). Fig. 4(a), (b) and Fig. 5(a), (b) show the MTF characteristics of the optical transfer functions F1 and F2 of the front blurred image and the rear blurred image. MTF is the amplitude characteristic of the optical transfer function, and is an expression used to grasp the spatial frequency characteristics.

[0027] First, referring to Figs. 4(a) and (b), the change in MTF when an aberration-imparting filter is applied will be described. Figs. 4(a) and (b) show MTF characteristics of the degradation characteristic F1 of the back blurred image and the degradation characteristic F2 of the front blurred image at a certain defocus amount. In Figs. 4(a) and (b), the horizontal axis shows spatial frequency, and the vertical axis shows MTF. 401 is the MTF of the degradation characteristic F1 of the back blurred image, and 400 is the MTF of the degradation characteristic F2 of the front blurred image. In the case of Fig. 4(a), an example is shown in which the back blurred image is sharp and the front blurred image is soft. Therefore, MTF 401 is higher in the high frequency range than MTF 400. Here, by applying an aberration-imparting filter F21, the MTFs can be made to match as shown in Fig. 4(b), and the state of the front and rear blurs can be matched.

[0028] Next, referring to Fig. 5(a) and (b), the change in MTF in the defocus direction when an aberration-imparting filter is applied will be described. Fig. 5(a) and (b) show the MTF characteristics of the degradation characteristic F1 of the back blur and the degradation characteristic F2 of the front blur, which correspond to the frequency 402 in Fig. 4(a). In Fig. 5(a) and (b), the horizontal axis indicates the defocus amount, and the vertical axis indicates the MTF.

[0029] Point 500 represents the in-focus position where the defocus amount is zero, and the positive direction of the defocus amount is the MTF corresponding to the foreground blurred image, and the negative direction is the MTF corresponding to the background blur. In the case of FIG. 5(a), as in FIG. 4(a), an example is shown in which the background blur is sharp and the foreground blurred image is soft. Therefore, the negative region has a higher frequency MTF than the positive region. The defocus amount corresponding to MTF 400 in FIG. 4(a) is 501, and the defocus amount corresponding to MTF 401 is 502.

[0030] Here, by applying the aberration imparting filter F21 while changing it according to the defocus amount, the MTF can be matched as shown in FIG. 5(b) to match the state of the front and rear blurs. That is, the blur size and the blur shape can be matched between the blurred image at the close position and the blurred image at the infinity position, which are equal in degree of deviation from the in-focus position (for example, the degree of defocus). It is also possible to make the front blurred image and the rear blurred image not completely matched (at least one of the blur size and the blur shape not matched). In general optical correction, if a correction is made to completely match, there is a drawback that artifacts due to image processing are generated or noise is amplified. Therefore, it is possible to actually weaken the correction by looking at the image quality after implementation. For example, it is possible to apply an LPF to the aberration imparting filters F12 and F21 to suppress shoots due to edge emphasis, or to weaken the gain of the contour emphasis so that the MTFs do not completely match in FIG. 4(a).

[0031] Furthermore, in areas where the amount of defocus near the in-focus point is relatively small and no significant blurring occurs, the shape of the blurring is not complex, and therefore it is possible to approximate the MTF characteristics simply by changing the contour emphasis and blurring processing in accordance with the optical information of the lens and the amount of defocusing, without requiring precise calculations of the aberration-imparting filter F21.

[0032] Next, a filter process (image processing method) for applying aberration to bring a front blurred image and a rear blurred image closer to each other will be described with reference to Fig. 6. Fig. 6 is a flowchart of the filter process. A program based on the flowchart of Fig. 6 is recorded in the non-volatile memory of the memory 105. This program is expanded in the RAM and executed by the CPU 108.

[0033] First, in step S1000, the CPU 108 reads image data from the imaging element 102, converts it into a digital signal by the A / D converter 103, then performs appropriate processing by the signal processing unit 104, and stores the processed image data in the memory 105. Next, in step S1001, the CPU 108 reads phase difference data from the imaging element 102, converts it into a digital signal by the A / D converter 103, then performs appropriate processing by the signal processing unit 104, and stores the processed phase difference data in the memory 105.

[0034] Next, in step S1002, the CPU 108 acquires lens data (optical information) related to the aperture diameter, focal length (zoom), and shooting distance (focus) of the imaging lens 101, and stores it in the memory 105. The CPU 108 also acquires aberration imparting data of the imaging lens 101, and stores it in the memory 105. Here, the aberration imparting data is an aberration imparting filter table configured as table data of the aperture diameter, focal length, shooting distance, and defocus amount, which is a filter kernel of the filter processing executed by the DSP 106. The filter kernel is, for example, design data obtained by calculating in advance the above-mentioned aberration imparting filter F21 or aberration imparting filter F12 based on the aperture diameter, focal length, shooting distance, and defocus amount.

[0035] This may be configured as data in the frequency domain, or may be configured as data in the spatial domain by performing an inverse Fourier transform on the aberration imparting filters F21 and F21. It is preferable to appropriately select depending on whether the filter processing of the DSP 106 is a spatial filter or a frequency filter. When designed as a spatial filter, for example, a filter kernel obtained by performing an inverse Fourier transform on the aberration imparting filter F21 or the aberration imparting filter F12 is held as each element of the table as 64 x 64 two-dimensional data.

[0036] The aberration imparting filter table may be configured with discrete thinned data for the aperture diameter, focal length, shooting distance, and defocus amount based on a certain table division number. For example, the aperture diameter may range from full aperture to small aperture. This range may be divided into four, and a filter kernel for a certain aperture diameter may be set as a division point, and the filter kernel may be calculated by interpolating between the division points using linear interpolation or the like. In this embodiment, the aberration imparting data is obtained from the imaging lens 101, but is not limited to this. For example, the aberration imparting data stored in a non-volatile memory of the memory 105 may be obtained.

[0037] Next, in step S1003, the CPU 108 obtains a block size from the memory 105. The block size is used to change the filter process executed by the DSP 106 for each block area. Next, in step S1004, the CPU 108 sets the block coordinates to an initial position in order to divide the image data into blocks and perform sequential filter processes. The initial position is, for example, the upper left corner of the image coordinate system, but is not limited to this.

[0038] Next, in step S1005, the CPU 108 determines a current block area based on the block size determined in step S1003 and the current block coordinates, and acquires phase difference data corresponding to the current block area from the memory 105. The CPU 108 also performs a predetermined process on the acquired phase difference data to calculate a representative value of the phase difference data in the current block area. The predetermined process is, for example, an arithmetic average, a weighted average, or a process in which data at a center position is used as a representative value, but is not limited to these. The CPU 108 also calculates a defocus amount by multiplying the representative value of the phase difference data by a predetermined coefficient for converting it into a defocus amount. The predetermined coefficient may be acquired from the imaging lens 101 when acquiring the lens data in step S1002, or may be stored in a non-volatile memory of the memory 105.

[0039] Next, in step S1006, the CPU 108 determines a filter kernel to be applied from the aberration applying filter table based on the aperture diameter, focal length, shooting distance, and defocus amount corresponding to the current block area acquired in step S1002. At this time, in the case of an aberration applying filter table in which data is discretized by having filter kernels only for specific division points as described in step S1002, a detailed filter kernel is calculated by interpolating between the division points.

[0040] Next, in step S1007, CPU 108 performs convolution filtering using DSP 106. DSP 106 applies convolution filtering to the current block area using the filter kernel determined in step S1006. Next, in step S1008, CPU 108 compares the current block coordinates with the end block coordinates calculated from the size of the image data, and determines whether filtering has been completed for all blocks. If it is determined that filtering has been completed for all blocks, the process proceeds to step S1010. On the other hand, if it is determined that filtering has not been completed for all blocks, the process proceeds to step S1009.

[0041] Next, in step S1009, CPU 108 increments the current block coordinate by one block. For example, the image is scanned from left to right and from top to bottom. The image is scanned horizontally from left to right, and when the right end is reached, the image moves to the left end of the block below and scans from left to right again. CPU 108 increments the block coordinate in this scanning method and performs filtering sequentially from the top left to the bottom right of the image. In step S1010, CPU 108 stores the image data after filtering in a non-volatile memory of memory 105.

[0042] Note that step S1007 may include a step of determining whether or not to perform filter processing (determination process). A flowchart in this case is shown in FIG. 7. FIG. 7 is a flowchart showing the determination process of whether or not to execute step S1007. A program based on the flowchart in FIG. 7 is recorded in the non-volatile memory of memory 105. This program is expanded in RAM and executed by CPU 108. The flowchart in FIG. 7 is executed after step S1006.

[0043] First, in step S2000, the CPU 108 acquires the kernel size of the filter processing to be executed in step S1007. As the distance from the in-focus position increases, the size of the blurred image increases, and therefore the kernel size when performing the filter processing also increases. As the kernel size increases, the processing load increases, so in order to perform the processing in real time, it is necessary to limit the processing to a certain size. The filter kernel size determined in step S1006 is linked to the aberration imparting filter table, and the filter kernel size is acquired by acquiring this table.

[0044] Next, in step S2001, CPU 108 compares the kernel size with the number of taps of the filter processing, and determines whether or not the kernel size exceeds the number of taps. If it is determined that the kernel size exceeds the number of taps, step S1007 is not executed, and the process proceeds to step S1008. On the other hand, if it is determined that the kernel size does not exceed the number of taps, the process proceeds to step S1007. This enables processing that takes into account the computing capacity of image processing device 100.

[0045] Although it has been considered that the aberration imparting filter is handled with a pre-calculated design value, data expressing the aberration characteristics of the point spread function or optical transfer function according to the distance from the in-focus position such as the defocus amount may be used instead. In this case, the aberration imparting filter table described in step S1002 is replaced with a point spread function table or optical transfer function table. Also, the CPU 108 directly calculates the filter kernel from the point spread function or optical transfer function to determine the aberration imparting filter table (F12 or F21).

[0046] In addition, the optical information acquired from the imaging lens 101 in step S1002 may include information on the aberration state of a lens device having an aberration variable mechanism. The filter kernel in the aberration imparting filter table may also be designed to include the aberration state of a lens device having an aberration variable mechanism. This makes it possible to reduce the difference between a front blurred image and a rear blurred image when the aberration is changed in a lens device having an aberration variable mechanism. In other words, it is possible to impart a soft focus effect or a bubble blur effect over the entire range.

[0047] As described above, the image processing device 100 performs filter processing on the image data using the lens data, the aberration imparting filter, and the defocus amount to bring the front blurred image and the rear blurred image closer together (reducing the difference between the front blurred image and the rear blurred image). This makes it possible to realize an expression in which the front and rear blurred images are similar to those of an old lens, even for a lens in which the front and rear blurred images are deviated due to other design factors, such as a design that prioritizes resolution.

[0048] <Second embodiment> Next, an image processing device 100a according to a second embodiment of the present invention will be described with reference to Fig. 8. Note that in this embodiment, differences from the first embodiment will be mainly described, and common descriptions will be omitted. This embodiment is similar to the first embodiment except for the configuration (method) for acquiring distance information for the focus in-focus position. In the first embodiment, a method for calculating distance information for the focus in-focus position based on an imaging surface phase difference detected from the imaging element 102 was described, but data for calculating the distance information may be input data from a sensor other than the imaging element.

[0049] Fig. 8 is a block diagram of the image processing device 100a. As shown in Fig. 8, in this embodiment, a distance sensor 700 is added to the image processing device 100 of the first embodiment. The distance sensor 700 acquires distance information such as information on the subject distance or a phase difference (information on the defocus amount based on the phase difference).

[0050] The distance sensor 700 is, for example, at least one of a phase difference detection sensor, a distance sensor, and a stereo camera, but is not limited thereto. The distance sensor 700 may be, for example, a laser range finder (LRF), a light detection and ranging (LiDAR), or the like, but is not limited thereto. The stereo camera may be one having a plurality of cameras. The distance information is multiplied by a predetermined coefficient in step S1005 to convert it into a defocus amount, so that the front and rear blurs can be matched in the same manner as in the first embodiment. In this way, in this embodiment, distance information for the focus position can be acquired from the distance sensor 700 other than the image sensor 102.

[0051] <Third embodiment> Next, an image processing device 100b according to a third embodiment of the present invention will be described with reference to FIG. 9. Note that in this embodiment, differences from the first or second embodiment will be mainly described, and common descriptions will be omitted. This embodiment is similar to the first embodiment except for the configuration (method) for acquiring image data, optical information of the lens, and distance information for the focus position. In the first embodiment, optical information of the lens is acquired from the imaging lens 101, and image data is acquired from the imaging element 102. Also, in the first embodiment, a method for calculating distance information for the focus position based on phase difference data detected from the imaging element 102 has been described, but these data may be input data from outside.

[0052] When completing the internal processing by the DSP 106 or the CPU 108, it may be difficult to execute a large number of complicated processes due to hardware constraints, etc. Therefore, in this embodiment, instead of using the imaging lens 101 and the imaging element 102, image data and necessary optical information of the lens, etc. are acquired, and filtering is performed on an application of a higher-precision image processing device such as a PC.

[0053] FIG. 9 is a block diagram of the image processing device 100b. As shown in FIG. 9, in this embodiment, the image processing device 100b has an input unit 800 instead of the imaging lens 101, the imaging element 102, and the A / D converter 103. The input unit 800 acquires optical information of the imaging lens from an external device and transfers it to the memory 105. The input unit 800 also acquires image data and distance information for the focus position from an external device, performs appropriate signal processing via the signal processing unit 104, and then transfers it to the memory 105. The optical information, image data, and distance information of the lens are, for example, recorded in a medium in association with the image data as metadata, and the input unit 800 can acquire them as input from the medium. Note that the configuration and processing other than those described above can match the state of front and rear blur in the same manner as in the first embodiment. In this way, in this embodiment, the optical information, image data, and distance information of the imaging lens can be acquired from the input unit 800.

[0054] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0055] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, and a program that are capable of appropriately expressing blur with high resolution.

[0056] The disclosure of each embodiment includes the following configurations and methods.

[0057] (Configuration 1) An acquisition means for acquiring distance information for a focus position; a filter processing unit that performs a filter process on image data from an image sensor using an aberration filter determined based on the distance information and optical information of an optical system, the aberration filter has a parameter for performing adjustment so that characteristics of a first blurred image formed on the image plane due to defocus on the closer side than the in-focus position and characteristics of a second blurred image formed on the image plane due to defocus on the infinity side than the in-focus position approach each other. (Configuration 2) 2. The image processing device according to configuration 1, wherein the optical information includes information regarding at least one of an aperture diameter, a zoom, and a focus. (Configuration 3) 3. The image processing device according to claim 1, wherein the optical information includes information regarding an aberration state on an image plane. (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the parameter is a parameter for performing sharpening processing or blurring processing so that the spatial frequency characteristics of the first blurred image and the spatial frequency characteristics of the second blurred image approach each other. (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the parameter is based on a composite function of an optical transfer function of the first blurred image and an inverse function of an optical transfer function of the second blurred image. (Configuration 6) The image processing device according to any one of configurations 1 to 5, characterized in that the parameter is a parameter for performing processing so that the blur size or blur shape of the first blurred image and the second blurred image, which have the same degree of defocus, approach each other. (Configuration 7) the acquiring means acquires an aberration characteristic based on the distance information and the optical information, 7. The image processing device according to any one of configurations 1 to 6, wherein the parameter is a parameter based on the aberration characteristic. (Configuration 8) 8. The image processing device according to configuration 7, wherein the acquisition means acquires the aberration characteristics from a lens device having the imaging optical system, an imaging device having the imaging element, or an external device. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the distance information is information regarding a defocus amount detected by an image plane phase difference detection method using the image sensor. (Configuration 10) 9. The image processing device according to any one of configurations 1 to 8, wherein the distance information is information regarding a defocus amount detected by a phase difference detection sensor. (Configuration 11) 9. The image processing device according to any one of configurations 1 to 8, wherein the distance information is information regarding a subject distance detected by a distance sensor. (Configuration 12) 12. The image processing device according to any one of configurations 1 to 11, wherein the filtering means performs the filtering process for each divided area of ​​the image data. (Configuration 13) the acquiring means acquires a number of taps of a filter kernel in the filtering process; 13. The image processing device according to any one of configurations 1 to 12, wherein the filter processing means does not perform the filter processing when it is determined that the number of taps is insufficient. (Configuration 14) 14. An imaging device comprising: the image processing device according to any one of configurations 1 to 13; and an imaging element. (Method 1) obtaining distance information for a focus position; and performing a filtering process on image data from an image sensor using an aberration filter determined based on the distance information and optical information of an optical system. an aberration filter having a parameter for performing adjustment so as to bring closer to each other characteristics of a first blurred image formed on an image plane due to defocusing on the closer side than the in-focus position and characteristics of a second blurred image formed on an image plane due to defocusing on the infinity side than the in-focus position. (Configuration 15) A program for causing a computer to execute the image processing method according to Method 1.

[0058] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0059] 100, 100a, 100b Image processing device 101 Imaging lens (optical system) 102 Image sensor 106 DSP (filter processing means) 108 CPU (acquisition method) 204 Front blurred image (first blurred image) 205 Back blurred image (second blurred image)

Claims

1. an acquisition means for acquiring distance information relative to a focus position; a filter processing means for performing a filtering process on image data obtained by using an image sensor, using an aberration filter determined based on the distance information and optical information of the optical system; an aberration filter having a parameter for performing adjustment so that the characteristics of a first blurred image formed on the image plane by defocusing closer to the in-focus position and the characteristics of a second blurred image formed on the image plane by defocusing on the infinity side of the in-focus position approach each other.

2. The image processing device according to claim 1 , wherein the optical information includes information relating to at least one of aperture diameter, zoom, and focus.

3. 2. The image processing apparatus according to claim 1, wherein the optical information includes information about an aberration state on an image plane.

4. 2. The image processing device according to claim 1, wherein the parameter is a parameter for performing a sharpening process or a blurring process so that the spatial frequency characteristics of the first blurred image and the spatial frequency characteristics of the second blurred image approach each other.

5. 2. The image processing apparatus according to claim 1, wherein the parameter is based on a composite function of an optical transfer function of the first blurred image and an inverse function of an optical transfer function of the second blurred image.

6. 2. The image processing apparatus according to claim 1, wherein the parameter is a parameter for performing processing so that the first blurred image and the second blurred image, which have the same degree of defocus, have blur sizes or blur shapes that approach each other.

7. the acquiring means acquires the aberration characteristics based on the distance information and the optical information, 2. The image processing apparatus according to claim 1, wherein the parameter is a parameter based on the aberration characteristic.

8. 8. The image processing apparatus according to claim 7, wherein the acquisition unit acquires the aberration characteristics from a lens device having the imaging optical system, an imaging device having the imaging element, or an external device.

9. 2. The image processing apparatus according to claim 1, wherein the distance information is information about a defocus amount detected by an image plane phase difference detection method using the image sensor.

10. 2. The image processing apparatus according to claim 1, wherein the distance information is information about a defocus amount detected by a phase difference detection sensor.

11. 2. The image processing device according to claim 1, wherein the distance information is information about a subject distance detected by a distance sensor.

12. 2. The image processing apparatus according to claim 1, wherein the filtering means performs the filtering for each of the divided regions of the image data.

13. the obtaining means obtains the number of taps of a filter kernel in the filtering process; 2. The image processing apparatus according to claim 1, wherein the filter processing means does not perform the filter processing when it is determined that the number of taps is insufficient.

14. An imaging device comprising: the image processing device according to claim 1; and an imaging element.

15. obtaining distance information for a focus position; and performing a filtering process on image data obtained using an image sensor using an aberration filter determined based on the distance information and optical information of the optical system, an aberration filter having a parameter for performing adjustment so that the characteristics of a first blurred image formed on the image plane by defocusing closer to the in-focus position and the characteristics of a second blurred image formed on the image plane by defocusing on the infinity side of the in-focus position approach each other.

16. A program causing a computer to execute the image processing method according to claim 15.

Images