Image processing device, image processing method, imaging device, and program

Abstract

To provide an image processing apparatus that can improve convenience by adjusting and displaying the positional relationship between an image and related information related to the image.SOLUTION: An imaging apparatus acquires an image and related information in a depth direction of a subject related to the image and performs image processing. A distance information generation unit 300 generates an additional information distribution related to the image (defocus map and distance map). A distortion and shake correction unit 301 executes deformation processing on the acquired image and executes deformation processing on the additional information distribution according to the deformation of the image. A color information conversion processing unit 303 performs conversion processing from the additional information distribution to color information. A superimposition processing unit 304 outputs, to a display, an image obtained by superimposing image information obtained through the conversion from the additional information distribution to the color information on a picked-up image after the deformation processing.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technology for presenting an image and its related information, such as depth information, to a user to assist in adjusting focus and depth. [Background technology]

[0002] In subject detection in an imaging device, image processing is performed to indicate to the user whether a specific subject is in focus, for example. Patent Document 1 discloses a technique known as peaking, which emphasizes the outline of a subject in focus. Patent Document 2 also discloses a technique for converting a color image to monochrome and then coloring the subject image in a color corresponding to the subject distance during manual focus operation, so that the user can intuitively grasp the sense of distance to the focused subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-73027 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-135812 [Patent Document 3] Japanese Patent Application Laid-Open No. 2014-93714 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology disclosed in Patent Document 1, when focusing on a subject, the closer the focus is, the more the subject's outline is displayed, but the user cannot grasp the depth of field of the shooting scene. Therefore, there is a need for improved convenience when adjusting the depth of field.

[0005] Furthermore, in the prior art disclosed in Patent Document 2, an image is generated and displayed by coloring an area determined based on a distance image in a monochrome image in which high-frequency components are emphasized. However, since no consideration is given to misalignment between the monochrome image and the distance image, if there is a misalignment between the two images, coloring cannot be performed on the correct area. An object of the present invention is to provide an image processing device that can improve convenience by presenting an image and related information related to the image in a consistent positional relationship. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present invention includes: of An image processing device that acquires and processes related information, an acquisition means for acquiring a distribution of additional information related to the image as the related information; The image and related information As a transformation process, aberration correction and image blur correction are performed. a transformation means for transforming the image based on the related information; Exit and output means for outputting an output signal to the deformation means, performing the aberration correction and image blur correction on the image using an interpolation method that calculates a new pixel value by weighting and combining the value of a pixel of interest and the values ​​of its surrounding pixels; For the image The aforementioned In response to the transformation process The aberration correction and image blur correction are performed on the distribution of the additional information using a nearest neighbor interpolation method that calculates a new pixel value from the value of the pixel of interest or its surrounding pixels, and the output unit outputs the related information after the transformation process and the image after the transformation process. It is characterized by: [Effects of the Invention]

[0007] According to the image processing device of the present invention, an image processing device capable of improving convenience can be provided by presenting an image and related information related to the image in a coordinated positional relationship. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a functional configuration of an imaging apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating a configuration of an imaging element included in an imaging unit according to the embodiment. [Figure 3] FIG. 2 is a block diagram showing a configuration of an image processing unit according to the embodiment. [Figure 4] FIG. 10 is an explanatory diagram of an image division process. [Figure 5] FIG. 10 is an explanatory diagram of derivation of a defocus amount. [Figure 6] 4 is a flowchart illustrating image processing in the first embodiment. [Figure 7] FIG. 2 is an explanatory diagram of a subject distribution in an imaging range in the first embodiment. [Figure 8] FIG. 4 is a diagram illustrating a defocus map according to the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating the results after distortion and blur correction according to the first embodiment. [Figure 10] FIG. 4 is an explanatory diagram of a conversion process from a defocus amount to α information according to the first embodiment. [Figure 11] FIG. 2 is an explanatory diagram of an example of an image display according to the first embodiment. [Figure 12] FIG. 10 is a block diagram showing the configuration of a distance information generating unit according to a second embodiment. [Figure 13] 10 is a flowchart illustrating a process in a second embodiment. [Figure 14] FIG. 10 is a schematic diagram illustrating the processing in the second embodiment. [Figure 15] FIG. 10 is a schematic diagram illustrating an example of a filter kernel shape. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiments of the present invention will be described in detail below with reference to the drawings. In the embodiments, an example of an image processing device will be described, which is an application example to an imaging device capable of acquiring depth information and distance information of a subject in an image. Depth information is information corresponding to the distance distribution of a subject in the depth direction (depth direction) within an imaging range. The present invention is applicable to any device capable of acquiring a captured image and distance information related to the imaging range of the captured image. Distance information is two-dimensional information indicating the distribution of the defocus amount of an image at each pixel of the captured image. As an example, a distribution of values ​​obtained by normalizing the defocus amount by the focal depth (e.g., 1Fδ, where F represents the aperture value and δ represents the allowable circle of confusion diameter) will be described. Here, for the aperture value F, a fixed value across the entire surface, with the aperture value near the center of the image height as the representative value, may be applied. Alternatively, a distribution of aperture values ​​that takes into account the fact that aperture values ​​at peripheral image heights become darker due to vignetting in the imaging optical system may be applied. Hereinafter, a distribution based on the defocus amount will be referred to as a "defocus map."

[0010] The distance information used in the application of the present invention may be information corresponding to the depth-direction distance distribution of the subject within the imaging range. For example, it is possible to use distribution information of the defocus amount before normalization by the focal depth or a depth map showing the subject distance corresponding to each pixel. It may also be two-dimensional information showing the phase difference used to derive the defocus amount. This phase difference corresponds to the relative image shift amount between different viewpoints. It is also possible to use a distance map converted into actual distance information on the subject side via the position of the focus lens of the imaging optical system. In other words, any information can be used as the distance information as long as it shows a change according to the depth-direction distance distribution.

[0011] [First Example] FIG. 1 is a block diagram showing the functional configuration of a digital camera (hereinafter simply referred to as "camera") 100 according to this embodiment. The camera 100 is an example of an imaging device equipped with an image processing device. The image processing device processes an image on which a defocus map is superimposed (hereinafter referred to as a "map-superimposed image"). For example, there are embodiments in which the image processing device performs display processing of the map-superimposed image, and embodiments in which the map-superimposed image processed by the image processing device is output to an external device, and the external device displays the map-superimposed image.

[0012] The system control unit 101 controls each component included in the camera 100. The system control unit 101 includes, for example, a CPU (Central Processing Unit) and reads an operating program from a ROM (Read Only Memory) 102, loads it into a RAM (Random Access Memory) 103, and executes it. The ROM 102 is a rewritable non-volatile memory such as a flash ROM. The ROM 102 stores not only the operating program but also parameters necessary for the operation of each component of the camera 100. The RAM 103 is a rewritable volatile memory. The RAM 103 is used not only as an area for loading the operating program but also as a temporary storage area for intermediate data output by the operation of each component of the camera 100. In this embodiment, the system control unit 101 and an image processing unit 107 (described later) use the RAM 103 as a working memory.

[0013] The optical system 104 is an imaging optical system that focuses light from a subject onto the imaging unit 105. The optical system 104 has, for example, a fixed lens, a variable magnification lens that changes the focal length, and a focus lens that adjusts the focus. The optical system 104 has an aperture, and the aperture adjusts the aperture diameter of the optical system 104 to adjust the amount of light during shooting.

[0014] The imaging unit 105 includes an imaging element such as a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor. The imaging unit 105 performs photoelectric conversion on an optical image formed on the imaging surface of the imaging element by the optical system 104, and outputs an analog image signal to the A / D conversion unit 106. The A / D conversion unit 106 performs A / D conversion processing on the input analog image signal, and outputs digital image data (hereinafter simply referred to as image data) to the RAM 103 for storage.

[0015] The image processing unit 107 performs various image processing operations on the digital image data stored in the RAM 103. Specifically, when Bayer RGB image data is input, the image processing unit 107 performs synchronization processing to generate R, G, and B color signals. Next, the image processing unit 107 adjusts the white balance by performing gain multiplication processing on the R, G, and B color signals based on the gain value for white balance adjustment. Processing is performed to generate a luminance signal Y from the RGB signal, and various processing operations, such as edge enhancement and luminance gamma correction, are performed on the luminance signal Y to output the image signal. Furthermore, matrix operations and the like are performed on the R, G, and B color signals to convert them to the desired color balance and perform gamma correction, and then color difference signals UV are generated. The image processing unit 107 records the image data after image processing on the recording medium 108. The image processing unit 107 also includes multiple components (see FIG. 3) to realize the functions of the present invention. Details of the processing performed by each component will be described later.

[0016] The recording medium 108 is, for example, detachable from the camera 100, and is, for example, a memory card. The recording medium 108 records image data (captured image data) that has been subjected to image processing by the image processing unit 107, image signals (RAW image signals) that have been A / D converted by the A / D conversion unit 106, and the like.

[0017] The display unit 109 includes a display device such as an LCD (liquid crystal display device) and displays various information about the camera 100. The display unit 109 functions as a digital viewfinder by, for example, displaying A / D converted image data through the display while the image is being captured by the image capturing unit 105. The display unit 109 also displays on the screen a map-superimposed image onto which color information converted from the defocus information generated by the image processing unit 107 is superimposed.

[0018] The operation input unit 110 is used as a user input interface and includes a release switch, setting buttons, a mode setting dial, etc. When the operation input unit 110 detects an operation input by the user, it outputs a signal corresponding to the operation input to the system control unit 101. In addition, in an aspect in which the display unit 109 includes a touch panel sensor, the operation input unit 110 functions as an interface that detects touch operations on the screen of the display unit 109.

[0019] The functional block elements of the camera 100 are basically connected by a bus 111, and are configured to be able to send and receive signals to and from each other via the bus 111.

[0020] Next, the detailed configuration of the imaging element included in the imaging unit 105 will be described with reference to Fig. 2. Fig. 2(A) is a schematic diagram showing a configuration in which a plurality of pixels 200 are regularly arranged two-dimensionally. The Z direction perpendicular to the paper surface of Fig. 2(A) is defined as the optical axis direction, and two directions orthogonal to each other within the paper surface are defined as the X direction and the Y direction. The left-right direction is defined as the X direction, and the up-down direction is defined as the Y direction. The plurality of pixels 200 are arranged, for example, in a two-dimensional lattice pattern, but the arrangement is not limited to a lattice pattern, and other arrangements may also be adopted.

[0021] 2(B) is a schematic diagram showing one of the pixels. Each pixel 200 has a microlens 201 and a pair of photoelectric conversion units 202a and 202b. The first pupil-divided pixel is composed of the photoelectric conversion unit 202a, and the second pupil-divided pixel is composed of the photoelectric conversion unit 202b, and the image sensor has a ranging function using an imaging surface phase difference ranging method.

[0022] The pair of photoelectric conversion units 202a, 202b each have a rectangular shape with the Y direction as the longitudinal direction, and are formed to be the same size. In each pixel 200, the photoelectric conversion units 202a, 202b are arranged in line symmetry with the perpendicular bisector of the microlens 201 along the Y direction as the axis of symmetry. Note that the shape of the imaging surface of the pupil-divided pixel is not limited to this and may be any shape. Furthermore, the arrangement direction of the pupil-divided pixels is not limited to the X direction and may be the Y direction, etc., and application to other embodiments with three or more divisions is also possible.

[0023] The imaging unit 105 can acquire two images: an image A based on an image signal output from a first pupil-divided pixel and an image B based on an image signal output from a second pupil-divided pixel, which are included in all pixels of the imaging element. The images A and B have a parallax relationship that corresponds to the distance from the in-focus position. In other words, the images A and B are images from different viewpoints. More specifically, in each pixel 200, a pair of photoelectric conversion units 202a and 202b perform photoelectric conversion on different light beams incident via the microlens 201, depending on the amount of light received. That is, photoelectric conversion is performed on optical images based on light beams that have passed through different regions of the exit pupil of the optical system 104. Since the images A and B are generated based on light beams that have passed through different regions (pupil-divided regions) of the exit pupil, the subject is captured at shooting positions that are shifted by the difference in the center of gravity of the pupil-divided regions, resulting in parallax. In other words, the images A and B correspond to a group of images acquired by capturing the subject from different viewpoints.

[0024] In this embodiment, the image sensor of the image capturing unit 105 (FIG. 2) can capture images A and B used to derive the distance distribution of a subject within the image capturing range. However, the image capturing method for the images A and B may be, for example, a method of capturing the images A and B from a group of images captured by a plurality of image capturing devices installed at a distance equal to the baseline length. Alternatively, a method of capturing the images A and B from a group of images captured by a single image capturing device having a plurality of optical systems and image capturing units (such as a so-called binocular camera or a multi-lens camera) may be used.

[0025] 3 is a functional block diagram of the image processing unit 107. The image processing unit 107 includes a distance information generation unit 300, a distortion and shake correction unit 301, a resizing unit 302, a color information conversion processing unit 303, and a superimposition processing unit 304. In this embodiment, defocus map information indicating the distribution of defocus amounts is used as related information related to the captured image. Examples of transformation processing related to the image and related information are shown below: distortion correction, image shake correction, and resizing processing.

[0026] The distance information generation unit 300 generates a defocus map as additional information distribution data corresponding to the image related to the image signal by analyzing the image signal acquired by the imaging unit 105. The distortion / blur correction unit 301 corrects image distortion caused by the characteristics of the optical system 104 and image blur caused by camera shake or the like for the display image to be displayed on the display unit 109 and the defocus map.

[0027] A resizing unit 302 resizes the defocus map to match the resolution of the image for display. A color information conversion unit 303 converts the defocus map values ​​into color information. A superimposition unit 304 superimposes the color information from the color information conversion unit 303 onto the image for display, generating a map-superimposed image.

[0028] The calculation process for the defocus amount will be described with reference to Figures 4 and 5. The distance information generation unit 300 generates a defocus map as information indicating the distance distribution of the subject in the depth direction of the imaging range. The defocus map includes information on the defocus amount of each subject image included in the captured image, and has a pixel structure corresponding to the captured image. The defocus amount can be derived based on the acquired parallax information, that is, based on image A and image B, which are a group of images having parallax.

[0029] FIG. 4 is an explanatory diagram of image division processing. For example, processing is performed to divide an image (images A and B) 700 into minute blocks 701 indicated by dashed lines. When each pixel of the target image A is taken as a pixel of interest, the minute blocks 701 are set in an area of ​​a predetermined size centered on the pixel of interest. Hereinafter, the minute blocks 701 are set as square areas of m×m pixels centered on the pixel of interest, but the shape and size of the minute blocks 701 can be set arbitrarily. Furthermore, a minute block 701 is set for each pixel of interest, and minute blocks 701 may overlap between different pixels of interest.

[0030] When minute blocks 701 are set for each pixel in images A and B, correlation calculation processing is performed for each pixel (pixel of interest) between the two images, and the amount of image shift (image shift amount) of the image contained in the minute block 701 corresponding to that pixel is derived. For example, assume that the number of data pieces (number of pixels) of a pair of minute blocks 701 defined for pixels of interest at the same position in images A and B is m. The pixel data of the pair of minute blocks 701 are represented as E(1) to E(m) and F(1) to F(m), respectively, and the amount of data shift is represented as k, with the unit being pixel. k is an integer value. When the amount of correlation is represented as C(k), the correlation calculation is performed using the following equation (1). C(k)=Σ|E(n)-F(n+k)| ···(1)

[0031] The Σ operation in equation (1) is performed on variable n, where n and n+k are limited to the range of 1 to m. The shift amount k is the relative shift amount in units of the detection pitch of a pair of image data. In this way, the correlation amount is derived for a pair of pupil division images (a pair of minute blocks 701) related to one pixel of interest. A specific example is shown in FIG. 5.

[0032] In Figure 5, the horizontal axis represents the shift amount k, and the vertical axis represents the correlation amount C(k). The shift amount k and the correlation amount C(k) have a discrete relationship. In this case, the correlation amount C(k) is smallest at the image shift amount with the highest correlation, so the shift amount x can be derived using the three-point interpolation method shown in equations (2) to (5) below. While the shift amount k is discrete, the shift amount x is the amount that gives the smallest value C(x) for the continuous correlation amount. x=kj+D / SLOP (2) C(x)=C(kj)-|D| (3) D={C(kj-1)-C(kj+1)} / 2 ···(4) SLOP=MAX{C(kj+1)-C(kj),C(kj-1)-C(kj)} ···(5) Here, kj is the shift amount k at which the discrete correlation amount C(k) is minimized. The shift amount x calculated in this way is included in the distance information as the image shift amount at one pixel of interest. The image shift amount is expressed in units of pixels.

[0033] Therefore, the defocus amount (denoted as DEF) at each pixel of interest can be derived from the following equation (6) using the image shift amount x. DEF=KX PY x (6) Here, PY is the pixel pitch of the image sensor (the distance between the pixels that make up the image sensor, in mm / pixel). KX is a conversion coefficient determined by the magnitude of the opening angle of the center of gravity of the light beams that pass through a pair of ranging pupils. Note that the magnitude of the opening angle of the center of gravity of the light beams that pass through a pair of ranging pupils changes depending on the size of the lens aperture (F-number), and therefore is determined based on the settings information at the time of image capture.

[0034] The distance information generator 300 derives the defocus amount of the subject at each pixel in the captured image by repeatedly shifting the pixel position of interest by one pixel at a time. After the defocus amount for each pixel is derived, a value normalized by the focal depth is calculated, and a defocus map, which is two-dimensional information with the same structure as the captured image, is generated using this value as the pixel value. That is, the defocus amount varies depending on the amount of deviation in the depth direction from the in-focus subject distance in the captured image. Therefore, the defocus map contains information equivalent to the depth distribution of the subject at the time of image capture. Furthermore, normalization by the focal depth makes it possible to grasp the depth change in the depth direction. The defocus map can also be used to identify the in-focus area (the area in focus) in the captured image.

[0035] Control of the shooting operation will be described with reference to Fig. 6. With camera 100, the user adjusts the depth while viewing the map superimposed image on the display screen and then shoots. Fig. 6 is a flowchart illustrating the processing executed by camera 100. The following processing is realized by the CPU of system control unit 101 reading out a program stored in, for example, ROM 102, expanding it into RAM 103, and executing it.

[0036] When the camera 100 is powered on, image data acquisition processing is executed in S401. In order to display the state of the imaging range on the display unit 109, the imaging unit 105 acquires image data under the control of the system control unit 101. The acquired image data includes an image A related to the first pupil-dividing pixel, an image B related to the second pupil-dividing pixel, and an added image of the images A and B (image A+B). The added image is an image corresponding to a state without pupil division, and is used as a display image. A specific example will be described later using FIG. 7.

[0037] In S402, the distance information generation unit 300 generates a defocus map corresponding to the image for display based on the images A and B acquired in S401. In S403, the distortion / shake correction unit 301 performs distortion correction and electronic image shake correction on the image for display acquired in S401 and the defocus map generated in S402. Methods for distortion correction and image shake correction are well known, and specifically, the technology disclosed in Patent Document 3 can be applied.

[0038] In S404, the resizing processing unit 302 performs resizing processing so that the resolution of the defocus map corrected in S403 becomes the same as the resolution of the image for display. In S405, the color information conversion processing unit 303 converts the value of the defocus map resized in S404 into color information that is easy for the user to view.

[0039] In S406, the superimposition processing unit 304 transparently superimposes the color information converted from the defocus amount in S405 onto the display image corrected in S403. In S407, the system control unit 101 controls the display unit 109 to display the image generated by the image processing unit 107 in S406.

[0040] In S408, processing for changing the aperture value of the optical system 104 is performed. For example, suppose that the user looks at the image displayed on the screen of the display unit 109 and realizes that the human subject is not within the depth of field. In this case, the user performs an operation to change the aperture value of the optical system 104 to a smaller aperture value via the operation input unit 110. The system control unit 101 receives the operation signal at this time and controls the driving of the aperture of the optical system 104 in accordance with the operation instruction.

[0041] In S409, the system control unit 101 determines whether or not the user has issued an instruction to take a photograph via the operation input unit 110. If it is determined that an instruction to take a photograph has been issued, the process proceeds to S410. On the other hand, if it is determined that an instruction to take a photograph has not been issued, the process returns to S401, where image data is acquired and the process of updating the image displayed on the display unit 109 continues. In S410, the system control unit 101 controls the photographing operation, and then ends the series of processes.

[0042] The processing shown in FIG. 6 will be described in detail with reference to FIGS. 7 to 11. FIG. 7 is a diagram illustrating the distribution of subjects within an imaging range 500. The imaging range 500 includes a person subject 501, a person subject 502 standing at a greater distance from the camera 100 than the person subject 501, and a horizon 503. As shown exaggeratedly in FIG. 7, the image of the horizon 503 is distorted due to distortion caused by the optical system 104. If the image were displayed as is, it would appear unnatural because it would differ from the actual scene. Furthermore, it is assumed that the person subject 501 is in focus within the depth of field, while the person subject 502 is slightly out of focus and out of focus. In this embodiment, the aperture value is changed in S408 of FIG. 6, and the depth range of the person subject 502 is also adjusted so that it is within the depth of field before being photographed. The image size is 6000 × 4000 pixels.

[0043] FIG. 8 is a diagram showing a defocus map corresponding to an image for display. The defocus map 800 is generated by the distance information generation unit 300 under the control of the system control unit 101 in S402 of FIG. 6. The defocus map 800 shows an example in which the defocus amount normalized by the focal depth is converted into a grayscale value and visualized. The shorter the distance from the camera 100 to the subject (subject distance), the closer the pixel value is to white (higher pixel value), and the longer the subject distance, the closer the pixel value is to black (lower pixel value). The image is expressed using a continuous grayscale so that an in-focus area (in-focus area) is displayed in 15% gray. For example, the in-focus person subject 501 is displayed in 15% gray. The back-focused person subject 502 is displayed in 35% gray. In addition, in the defocus map 800, the area corresponding to the horizon 503 is distorted due to distortion caused by the optical system 104. It should be noted that the size of the minute block when calculating the defocus amount is 10×10 pixels, and the size of the defocus map 800 is 600×400 pixels.

[0044] FIG. 9 is a schematic diagram showing a display image and a defocus map after distortion and blur correction. The distortion and blur correction unit 301 performs distortion correction and electronic image blur correction (hereinafter simply referred to as image blur correction) under the control of the system control unit 101 in S403 of FIG. 6. FIG. 9(A) shows a display image 900 after correction, and FIG. 9(B) shows a defocus map 901 after correction. It can be seen that deformation due to distortion has been corrected. In conventional peaking display, edges in the display image are extracted and highlighted, so distortion correction and image blur correction only need to be performed on the display image. However, when a defocus map is superimposed on the display image (displaying a map-superimposed image), distortion correction and image blur correction must also be performed on the defocus map based on the correction process performed on the display image. This is to prevent misalignment between the defocus map and the display image. By superimposing the defocus map in which the positional deviation has been corrected on the display image, it is possible to superimpose distance information in the correct area while eliminating any unnatural appearance.

[0045] Regarding the pixel interpolation calculation method used when performing distortion correction and image blur correction, it is advisable to use different methods for the image to be displayed and the defocus map. Specifically, when correcting the image to be displayed, an interpolation method is selected that performs weighted synthesis (weighted addition) by referencing the values ​​of surrounding pixels, such as bilinear interpolation that references a pixel of interest and its four neighboring pixels. The reason for this is that users perceive better image quality when pixel values ​​are processed so that they change smoothly in the process of creating an image to be viewed by the user.

[0046] On the other hand, when generating pixel values ​​of a defocus map by interpolation, there is a problem with performing bilinear interpolation in a so-called perspective conflict area, where pixels with a far-away subject distance exist around pixels with a close subject distance. If a pixel value indicating an intermediate distance (for example, a pixel value indicating that the image is in focus) occurs, there is a possibility that erroneous distance information will be displayed. Therefore, the nearest neighbor interpolation method is selected as the calculation method for interpolation of the defocus map, which is an image of evaluation values.

[0047] In this embodiment, distortion correction and image blur correction are performed on a defocus map. For example, assume a configuration in which distortion correction and image blur correction are performed on the images A and B referenced in S402 of FIG. 6 . In this case, if a roll-direction blur component is included, the direction of parallax will be shifted from the pupil division direction (the horizontal direction in this embodiment) due to image blur correction. Therefore, the calculation result of Equation (1) for calculating the correlation amount may deviate depending on whether or not correction is performed. This effect is particularly significant when the subject image has diagonal lines, potentially reducing the accuracy of the calculated defocus map. Furthermore, to reduce the computational load, the system is configured to select not to perform correction processing if the distortion correction amount and image blur correction amount in the captured image are smaller than a predetermined amount (threshold value).

[0048] In S404 of FIG. 6, under the control of the system control unit 101, the resizing unit 302 performs resizing so that the resolution of the defocus map corrected in S403 becomes the same as the resolution of the display image. In this embodiment, the size of the defocus map is 600 × 400 pixels, and the size of the display image is 6000 × 4000 pixels. Therefore, the defocus map is enlarged by 10 times in both the horizontal and vertical directions. At this time, as in S403, the nearest neighbor interpolation method is selected as the pixel interpolation calculation method for the enlargement process.

[0049] Regarding the processing order of distortion correction, image stabilization, and resizing, the resizing process in S404 in Fig. 6 is performed after the distortion correction and image stabilization in S403. Unlike images displayed for viewing, the resolution of a defocus map can be reduced by setting small blocks. Reducing the resolution reduces the computational load and scale required for distortion correction and image stabilization.

[0050] Next, the color information conversion process performed by the color information conversion processing unit 303 under the control of the system control unit 101 at S405 in FIG. 6 will be described. In the process of converting the defocus amount to color information, the grayscale value representing the defocus amount is converted to a color value (color difference signal UV) using a lookup table or similar. The color values ​​are converted to, for example, a color contour, where the grayscale value is blue, light blue, green, yellow, and red in ascending order. The color contour color scheme can be selected and set according to the user's preferences and ease of viewing. For example, it is also possible to select a color contour where the grayscale value is blue, light blue, green, yellow, and red in ascending order. That is, in this embodiment, the type of color conversion style for the distance information distribution can be changed or adjusted. By representing the distance information distribution in color in this way, subtle differences in blur that are difficult to distinguish on a relatively small monitor such as the LCD monitor of the camera 100 can be visually more easily distinguished. This improves the user's convenience when adjusting the depth and focus position.

[0051] Furthermore, when converting the grayscale value representing the defocus amount into color information, the in-focus area shown in FIG. 9B as 15% gray may be converted to a single color, such as green. Conventional peaking displays are highly dependent on edge strength in the display image, potentially causing the peaking display to react to strong edges, such as building boundaries, even in defocused areas. In contrast, this embodiment uses a defocus amount based on the parallax amount, reducing dependency on edge strength and allowing the user to more accurately confirm focus. Furthermore, as with the color contours described above, focus status can be expressed by color, improving user convenience in terms of visibility. The data conversion required to indicate the degree of focus based on color intensity (display density) is described below with reference to FIG. 10.

[0052] FIG. 10 is a diagram for explaining the conversion from defocus amount to α information. The horizontal axis represents the defocus amount, and the vertical axis represents the α value. The α information determines the density on the display, and the closer the α value is to 1.0, the darker the color will be when coloring is performed. FIG. 10(A) shows an example of a triangular graph, and FIG. 10(B) shows an example of a trapezoidal graph.

[0053] In Figure 10(A), the α value is zero when the defocus amount is less than -3Fδ, and increases linearly as the defocus amount increases when the defocus amount is greater than -3Fδ and less than 0Fδ. The α value corresponding to the defocus amount (0Fδ) shown in 15% gray is 1.0. When the defocus amount is greater than 0Fδ and less than 3Fδ, the α value decreases linearly as the defocus amount increases. When the defocus amount is greater than 3Fδ, the α value is zero.

[0054] In Figure 10(B), the α value is zero when the defocus amount is less than -5Fδ, and increases linearly as the defocus amount increases when the defocus amount is greater than -5Fδ and less than -3Fδ. The α value is 1.0 when the defocus amount is greater than -3Fδ and less than 3Fδ. The α value decreases linearly as the defocus amount increases when the defocus amount is greater than 3Fδ and less than 5Fδ. The α value is zero when the defocus amount is greater than 5Fδ.

[0055] As described above, the range of the defocus amount that is colored darkly can be adjusted by the user to a desired range. The user can set the range that is considered to be in focus to any range by widening or narrowing it (the width of the upper side of the trapezoid in FIG. 10) according to the purpose, thereby improving convenience. Note that the user can specify in advance whether the display format is color contour or one-color display by setting the camera 100.

[0056] In S406 of FIG. 6, under the control of the system control unit 101, the superimposition processing unit 304 transparently superimposes the color information converted from the defocus amount in S405 onto the display image that has been subjected to the correction processing. Specifically, if the display format is color contour, the color difference signals UV of the display image are replaced with color difference signals UV of the color contour. At this time, it is preferable to convert the value range of the luminance signal Y in advance so that it is narrowed, so that the hue after superimposition does not change significantly depending on the value of the luminance signal Y in the display image. Specifically, assuming that the original value range of the luminance signal Y is 8 bits from 0 to 255, the conversion is performed so that the value range becomes, for example, 20 to 235.

[0057] FIG. 11 is a schematic diagram showing a map superimposed image 1100 in which the defocus map of FIG. 9(B) is superimposed on the display image of FIG. 9(A). For convenience, FIG. 11 shows a gray display, but the actual display is in color. Note that when the display format is a one-color display, weighted addition of the display image and the one-color YUV signal value is performed based on the α value in S405 of FIG. 6. The defocus map information is transparent, allowing the user to visually recognize the display image on which the superimposition processing has been performed, making it easy to grasp the correspondence with the depth and focus information. This allows the user to make adjustments while checking the depth and focus state in the shooting scene.

[0058] Alternatively, an edge in the image for display may be extracted and combined with the color conversion information. In this configuration, the color-removed edge in the image for display is used, which has the advantage that there is no mixing of the color information converted from the defocus amount with the color in the image for display, making the defocus map information easier to see.

[0059] In this embodiment, an example is shown in which the defocus amount is converted into color information to improve user visibility, but to reduce the processing load, a configuration is also possible in which the defocus amount is displayed in grayscale. Even in this case, subtle differences in blur can be more easily distinguished on the LCD monitor of the camera 100. This can improve convenience when the user adjusts the depth and focus position.

[0060] In S407 of FIG. 6, the system control unit 101 controls the display unit 109 to display an image for display on which a defocus map generated by the image processing unit 107 is transparently superimposed. The user can check the map-superimposed image displayed on the screen of the display unit 109. By correcting distortion aberrations and the like, the unnatural appearance is eliminated, and the depth and focus state can be grasped with good visibility based on the distance information superimposed with reduced positional deviation. For example, in S408 of FIG. 6, the user looks at the image displayed on the display unit 109 and performs an operation to change the aperture value of the optical system 104 to a smaller aperture so that the desired human subject 502 (FIG. 7) is within the depth. The system control unit 101 drives the aperture of the optical system 104 in accordance with the user's operation instruction.

[0061] In S409 of FIG. 6, the system control unit 101 checks whether the user has issued a command to take a photograph via the operation input unit 110. If a command to take a photograph has been issued, the process proceeds to S410. In S408, the aperture value is changed to a smaller aperture, deepening the depth of field, and the person subject 502 (FIG. 7), which was in back focus, now falls within the depth of field. An image is displayed in which the color indicating the area in focus (in-focus area) has changed. In this way, the user can adjust the aperture value while viewing the displayed map-overlaid image. The user checks that the desired color is superimposed on the image of the person subject 502, i.e., that the person subject 502 is within the depth of field, and then issues a command to the camera 100 to take a photograph.

[0062] 6, the system control unit 101 controls the shooting operation in accordance with a shooting instruction from the user via the operation input unit 110, and records the captured image data on the recording medium 108. In this way, the user can determine optimal shooting settings while checking the depth of field. This can suppress noise degradation caused by an increase in ISO sensitivity that occurs when the aperture is set too small, and subject blur caused by an increase in exposure time.

[0063] According to this embodiment, by presenting the user with a map superimposed image in which distance information with positional deviation corrected is superimposed on the display image, convenience when adjusting the focus position and depth can be further improved.

[0064] [Modification of the first embodiment] In the first embodiment, a defocus map is always superimposed on the display image. However, the present invention is not limited to this. In a modified example, the operation input unit 110 is provided with an operation device such as a push button. The defocus map is superimposed on the display image and displayed only while the user is operating the operation device. In other words, the map superimposed image display process may be performed for a limited display period. This configuration minimizes changes from the display format during shooting, and adaptively displays the map superimposed image only when the user wants to check the depth or focus, improving user convenience. In addition, in a modified example, the area superimposed on the display image is limited to, for example, the AF frame area. This allows the user to check the focus state of the area they are focusing on while minimizing changes from the conventional display format without map superimposition.

[0065] Furthermore, the image processing unit 107 of the modified example performs cyclic processing to average the defocus map with previously calculated maps in order to reduce fluctuations (flickering) in the defocus map along the time axis. This configuration reduces flickering in the map-overlapped image displayed on the display unit 109. This improves visibility when the user adjusts the depth or focus position. Furthermore, this configuration prevents abrupt color changes when the user changes the aperture value, resulting in a visually pleasing display. The cyclic processing involves storing image data in a dedicated RAM, which is configured to be processed before the distortion correction and image blur correction processes of S403 in FIG. 6 . The cyclic processing can be performed when the image data is read from the RAM 103 to the dedicated RAM. This reduces the number of accesses to the RAM 103, thereby reducing the overall system load on the camera 100. In this case, multiple distortion and blur correction units 301 are provided for different purposes. For example, one for a high-resolution display image and another for a low-resolution defocus map. By providing a dedicated RAM for a low-resolution defocus map, it is possible to achieve a balance between processing load and cost.

[0066] Furthermore, as a modified example, as a method that is independent of the subject distance, there is a configuration in which, for example, an optical flow that maps the distribution of motion vector information is superimposed on a display image. The motion vector information is information on the direction and amount of movement of the subject, and the movement (motion) of the subject includes movement in any direction within a two-dimensional plane and movement in the depth direction. Furthermore, in the first embodiment, a configuration for capturing still images was described. However, the present invention is not limited to this, and can also be applied to a configuration for capturing moving images. The above matters also apply to the embodiments described below.

[0067] [Second Example] A second embodiment of the present invention will be described with reference to Figures 12 to 15. In this embodiment, the same components as those in the first embodiment will be designated by the same reference numerals already used, and detailed descriptions thereof will be omitted, with differences being mainly described.

[0068] In this embodiment, the distance information generating unit 300 calculates a defocus map using a parallax image with a compressed resolution in the parallax direction to improve the calculation speed when generating defocus distribution information. Furthermore, during this process, a filter process is performed to suppress the occurrence of unnatural artifacts in the defocus map while retaining the defocus amount related to the main subject. To enable faster depth confirmation, the presentation interval of the map-overlaid image can be shortened, improving user convenience when adjusting depth.

[0069] The detailed configuration of the distance information generation unit 300 will be described with reference to Fig. 12. Fig. 12 is a functional block diagram of the distance information generation unit 300, which includes an image pre-processing unit 1200, a defocus amount derivation unit 1201, a kernel shape selection unit 1202, a filter processing unit 1203, and a map post-processing unit 1204.

[0070] The image preprocessing unit 1200 performs preprocessing on the acquired disparity information (a group of images having disparity). The preprocessing is a process performed before the calculation for deriving the defocus amount. The defocus amount derivation unit 1201 derives the defocus amount and generates a defocus map.

[0071] A kernel shape selection unit 1202 selects the shape of the filter kernel to be used in a filter processing unit 1203. The filter processing unit 1203 performs filtering on the defocus map generated by the defocus amount derivation unit 1201. As will be described later, the filter processing unit 1203 has the function of biasing the filter effect in a specific direction. A map post-processing unit 1204 performs post-processing of the defocus map according to the processing content of the image pre-processing unit 1200.

[0072] The processing flow in this embodiment will be described with reference to Figures 13 and 14. Figure 13 is a flowchart illustrating an example of the processing. Figure 14 is a schematic diagram showing a specific example of the processing. The following processing is realized by the CPU of the system control unit 101 reading out a program stored in, for example, ROM 102, expanding it in RAM 103, and executing it.

[0073] In S1301, the image preprocessing unit 1200 performs preprocessing for the defocus amount calculation on the acquired image 1400 ( FIG. 14 ) with parallax. In actual processing, there are viewpoint images corresponding to the number of parallaxes, but for simplicity, FIG. 14 shows only one image. The image 1400 includes an image of an object 1401 that is in focus and an image of an object 1402 that is out of focus and appears small. In this embodiment, resolution reduction processing is performed in the parallax direction to improve calculation speed. For example, if the size of the acquired image is 6000 × 4000 pixels, reduction processing is performed to one-third in the parallax direction, resulting in an image size of 2000 × 4000 pixels. This generates a reduced image 1410. In this embodiment, the reduction processing to one-third is performed by calculating the average value of three adjacent pixels. This is not limited to this, and the reduction processing to one-third may also be performed by thinning out two of the three adjacent pixels, for example.

[0074] In S1302, the defocus amount derivation unit 1201 derives a defocus amount using the reduced image 1410 acquired in S1301, and generates a defocus map 1420. The calculation details are as described in the first embodiment. The distribution of the defocus map 1420 includes a defocus amount 1421 related to the object 1401 and a defocus amount 1422 related to the object 1402. Noise 1423 also schematically indicates noise that occurs during calculation to derive the defocus amount. The size of the minute blocks used in calculating the defocus amount is 10×10 pixels, as in the first embodiment. On the other hand, the size of the defocus map corresponds to the image reduction and becomes 200×400 pixels, unlike in the first embodiment.

[0075] In S1303, the kernel shape selection unit 1202 selects the shape of the filter kernel to be used by the filter processing unit 1203 according to the processing content performed by the image preprocessing unit 1200. The filter processing unit 1203 performs filtering on the defocus map derived by the defocus amount derivation unit 1201. In this embodiment, median filtering is performed to remove noise. The kernel shape will be described with reference to FIG. 15.

[0076] FIG. 15 is a schematic diagram illustrating examples of kernel shapes (square, cross). For example, in conventional median filtering, a square median filter such as kernel 1501 is used. When median filtering is performed on a defocus map 1420 using kernel 1501, noise 1423 can be removed by median filtering, as shown in defocus map 1430 (FIG. 14). However, the defocus amount 1422 is calculated from data corresponding to an image in which a small object 1402 in the image is reduced in the parallax direction. Therefore, there is a concern that the defocus amount will be removed by the median filter, along with the noise 1423. Therefore, in this embodiment, median filtering is performed using a cross-shaped kernel 1502. As a result, a noise removal effect can be obtained without removing information from a long, narrow region (see defocus amount 1422) in the defocus map 1440.

[0077] The shape of the filter kernel is not limited to the cross-shaped kernel 1502 shown in this embodiment. For example, the aspect ratio may be changed depending on the degree of reduction. In other words, the filter characteristics are determined by the direction or reduction rate of the image reduction process. If the parallax direction is not horizontal, a kernel with a cross-shaped shape rotated to match the parallax direction can be used. Furthermore, the shape of the filter kernel does not need to be selected for each frame. For example, if the parameters in the process performed in S1301 of FIG. 13 are constant, a constant kernel shape can be used.

[0078] In S1304, the map post-processing unit 1204 performs post-processing on the defocus map 1440 after the filtering process. Enlargement processing is performed to restore the change in aspect ratio caused by the reduction in the parallax direction performed in S1301. Specifically, as in the defocus map 1450 of Fig. 14, enlargement processing is performed by a factor of three in the parallax direction (the horizontal direction in this example). As a result, the size of the generated defocus map becomes 600 x 400 pixels, similar to the first embodiment.

[0079] In this embodiment, it is possible to improve the calculation speed when generating a defocus map from acquired parallax information (a group of images having parallax). It is possible to perform filtering so that the occurrence of unnatural artifacts in the defocus map is suppressed and the defocus amount of the main subject remains.

[0080] [Modification of the second embodiment] In the second embodiment, a process for generating a defocus map using only a group of reduced images was described in order to improve calculation speed, but this is not limited to this. In a modified example, in order to improve the reliability of the map, a corresponding defocus map is derived for each different pre-processing step, and a filter process with different characteristics is performed. For example, a defocus map is derived for each of a first group of reduced images and a second group of unreduced images, and a filter process is performed on them. The two defocus maps are then merged to generate a final defocus map. In the filter process, a median filter process with a cross-shaped kernel is performed on the defocus map derived from the first group of images. On the other hand, a median filter process with a conventional square kernel is performed on the defocus map derived from the second group of images.

[0081] The merging process in this modified example can use the following method: For simplicity of notation, the defocus map derived from the non-reduced second group of images will be referred to as the "normal map" below, and the defocus map derived from the reduced first group of images will be referred to as the "reduced map" below.

[0082] As the defocus map, the values ​​of the normal map are basically used, but the values ​​of the reduced map are used depending on a predetermined condition (selection process). The predetermined condition is, for example, when the reliability of the normal map is less than a threshold value. The reliability can be calculated from the variance value of the small blocks of the image used to derive the defocus amount. Alternatively, the reliability may be calculated from the deviation value of the defocus amount relative to the surrounding area. Since the method of calculating the reliability is well known, a detailed explanation thereof will be omitted. In areas where the reliability of the normal map is low, the values ​​of the reduced map are used. Alternatively, in areas where the reliability of the normal map is low, only the defocus direction information indicated by the values ​​of the reduced map may be recorded.

[0083] Furthermore, for example, if there is a repetitive pattern in an out-of-focus area in the normal map, there is a possibility that a defocus amount indicating in-focus may be derived erroneously. Therefore, for areas where a defocus amount indicating out-of-focus has been derived, processing is performed to output the value of the reduced map.

[0084] Furthermore, for example, since a reduced image group is used to generate the reduced map, the boundaries of the regions may become rough. Therefore, in a region that is in focus in the reduced map and is derived as an out-of-focus defocus region in the normal map, a process is performed in which the reduced map is considered to be protruding and the value of the normal map is output.

[0085] In this modification, a parameter representing the size of a processing block (microblock) used to calculate a defocus amount is set as a parameter for image division processing. The characteristics of the filter processing are determined by the size or aspect ratio of the processing block. For example, the kernel shape selection unit 1202 selects a kernel shape according to the value of the parameter set by the defocus amount derivation unit 1201.

[0086] In the second embodiment, the process of deriving the defocus amount after reducing the image group is described, but this is not limited to this. In a modified example, a process of expanding the minute block (10 x 10 pixels) used when calculating the defocus amount by three times in the parallax direction is performed, resulting in a size of 30 x 10 pixels. In this way, the size of the defocus map becomes 200 x 400 pixels without reducing the image group, and the same process as in the second embodiment can be performed.

[0087] According to the above embodiment, by superimposing an image corresponding to distance information, etc., with the positional deviation corrected on the display image after capture and presenting it to the user, it is possible to improve convenience when adjusting the focus position and depth.

[0088] While the present invention has been described above as a preferred embodiment, it is not limited to the above embodiment and various modifications and variations are possible within the spirit and scope of the present invention. Specifically, while a digital camera, which is one example of an application of an image processing device, has been described, the present invention can also be applied to a computer or the like having the functionality of an image processing unit 107. Furthermore, while the above embodiment describes a defocus map generated based on a group of images that have a parallax relationship, this method is not limited to this method as long as it corresponds to the captured image and can acquire the distance distribution of the subject within the captured image range. One method for generating a defocus map is the DFD method, which derives the defocus amount from the correlation between two images with different focus and aperture values. DFD stands for "Depth From Defocus." The distance distribution of the subject can also be derived using related information related to the distance distribution obtained from a ranging sensor module, such as a TOF method. TOF stands for "Time Of Flight." Alternatively, related information related to the distance distribution can be acquired using a contrast ranging method based on the contrast information and evaluation value of the captured image. Regardless of the method used, related information on the distance distribution can reduce the positional deviation between the distance distribution and the captured image, thereby enabling a more accurate map-overlaid image to be displayed.

[0089] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0090] 100: digital camera, 101: system control unit, 104: optical system, 105: imaging unit, 107: image processing unit, 109: display unit, 110: operation input unit, 300: distance information generation unit, 301: distortion and shake correction unit, 302: resizing unit, 303: color information conversion unit, 304: superimposition unit

Claims

1. An image processing device that acquires and processes related information of an image and a subject in a depth direction related to the image, an acquisition means for acquiring a distribution of additional information related to the image as the related information; a transformation means for performing aberration correction and image blur correction as a transformation process on the image and related information; an output means for outputting an image based on the related information for the image, the deformation means performs the aberration correction and image blur correction on the image using an interpolation method that calculates a new pixel value by weighted synthesis of a value of a pixel of interest and values ​​of its surrounding pixels, and performs the aberration correction and image blur correction on the distribution of the additional information in response to the deformation processing on the image using a nearest neighbor interpolation method that calculates a new pixel value from the value of the pixel of interest or its surrounding pixels; The output means outputs the related information after the transformation process and the image after the transformation process.

1. An image processing device comprising:

2. the distribution of the additional information is a distribution of distance information related to the image, generating means for generating the distance information 2. The image processing device according to claim 1, wherein:

3. The distribution of distance information is a distribution of parallax information obtained from a group of images with different viewpoints, a distribution of contrast information obtained from a group of images with different focuses, or a distance distribution obtained by a TOF method.

3. The image processing device according to claim 2.

4. The parallax information is two-dimensional information having the same pixel structure as the image, and includes any one of a defocus amount in the captured image, a relative image shift amount between different viewpoints, and distance information.

4. The image processing device according to claim 3.

5. The output means changes the type of color conversion style for the distribution of the additional information or adjusts the density of the display.

5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. The method further includes a cyclic processing means for averaging the distribution of the acquired additional information with the additional information calculated in the past.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

7. The generating means a derivation means for deriving a distribution of the distance information; a filter processing means for performing a filter process on the distribution of the distance information; 5. The image processing device according to claim 2, wherein the image processing device is a computer.

8. The derivation means calculates the distribution of the distance information using an image whose resolution is compressed in the parallax direction.

8. The image processing device according to claim 7,

9. The filtering means has a median filter and uses a cross-shaped kernel to bias the filtering effect in a specific direction.

9. The image processing device according to claim 7 or 8.

10. a preprocessing means for preprocessing the image; A selection means is provided for selecting the shape of the kernel to be used by the filtering means based on the processing performed by the preprocessing means.

10. The image processing device according to claim 7, wherein the image processing device is a computer.

11. The preprocessing means performs a reduction process on the acquired image, The characteristics of the filtering means are determined by the direction or rate of reduction of the image.

11. The image processing device according to claim 10.

12. the distribution of distance information is a distribution of parallax information acquired from a group of images having different viewpoints, The characteristics of the filtering means are determined by the parallax direction. The image processing device according to claim 10 .

13. the derivation means sets parameters to be used in the segmentation process of the image; The characteristics of the filtering means are determined by the parameters. The image processing device according to claim 10 .

14. The parameter is a parameter representing the size of a block used to calculate the distribution of the distance information, and the characteristics of the filtering means are determined by the size or aspect ratio of the block.

14. The image processing device according to claim 13.

15. the preprocessing means performs first and second preprocessing on the image; The deriving means derives a first distribution of distance information from the image that has been subjected to the first preprocessing, derives a second distribution of distance information from the image that has been subjected to the second preprocessing, and integrates the first and second distributions of distance information.

15. The image processing device according to claim 10,

16. The preprocessing means performs a reduction process on the image in the first preprocessing and does not perform a reduction process on the image in the second preprocessing.

16. The image processing device according to claim 15,

17. The deriving means integrates the distribution of the distance information by selecting the first or second distance information based on a predetermined condition.

17. The image processing device according to claim 15 or 16.

18. a post-processing means for performing enlargement processing on the output of the filtering means; 12. The image processing device according to claim 11.

19. A cyclic processing means is provided for calculating an arithmetic mean of the acquired distribution of distance information and previously calculated distance information before the transformation processing.

19. The image processing device according to claim 18.

20. The transformation means performs a first transformation process on the image having a first resolution and a second transformation process on the related information having a second resolution smaller than the first resolution.

20. The image processing device according to claim 1, wherein the image processing device is a computer.

21. An image processing device according to any one of claims 1 to 20; an imaging element; An imaging device characterized by:

22. The imaging element includes a plurality of microlenses and a plurality of photoelectric conversion units corresponding to the respective microlenses, and outputs a plurality of image signals having different viewpoints.

22. The imaging device according to claim 21.

23. An image processing method executed by an image processing device that acquires and processes related information of an image and a subject in a depth direction related to the image, an acquisition step of acquiring a distribution of additional information related to the image as the related information; a transformation step of performing aberration correction and image blur correction as a transformation process on the image and related information; an output step of outputting an image based on the related information for the image, In the deformation step, the aberration correction and image blur correction are performed on the image using an interpolation method that calculates a new pixel value by weighted synthesis of a value of a pixel of interest and values ​​of its surrounding pixels, and in response to the deformation process being performed on the image, the aberration correction and image blur correction are performed on the distribution of the additional information using a nearest neighbor interpolation method that calculates a new pixel value from the value of the pixel of interest or its surrounding pixels, In the output step, a process of outputting the related information after the transformation process and the image after the transformation process is performed is performed. An image processing method comprising:

24. A computer is caused to function as each of the means of the image processing device according to any one of claims 1 to 20. A program characterized by:

Images