Image processing device and control method for image processing device

Abstract

To address a probability that, when the state of a lens, a barrel, or a sensor is changed due to a temperature change or posture change of an imaging device, optical characteristics such as a field curvature and a distortion aberration are changed to generate a ranging error in an overall screen.SOLUTION: An image processing device includes generation means for generating distance information using a plurality of image signals obtained from different view points, detection means for detecting a flat region in the image signals, and estimation means for estimating a correction map for correcting the distance information. The estimation means expresses the correction map by using a function in which position coordinates of an image are variables, and estimates the correction map on the basis of the distance information corresponding to the flat region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device for an imaging device such as a digital camera, a digital video camera, an in-vehicle sensor device, or a robot vision sensor device, which has a distance measurement function using an image, and a control method thereof. [Background technology]

[0002] There has been proposed an imaging device equipped with a distance measurement function that can detect distance information such as the defocus state of a subject and the distance from the imaging device to the subject based on image signals captured from different viewpoints (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2010 / 010707 Summary of the Invention [Problem to be solved by the invention]

[0004] In imaging devices with such distance measurement functions, changes in the state of the lens, lens barrel, and sensor due to temperature changes or changes in the orientation of the imaging device can change optical characteristics such as field curvature and distortion, potentially causing distance measurement errors across the entire screen. [Means for solving the problem]

[0005] In view of the above problems, the image processing device of the present invention comprises a generation means for generating distance information using a plurality of image signals captured from different viewpoints, a detection means for detecting planar areas in the image signals, and an estimation means for estimating a correction map for correcting the distance information, wherein the estimation means represents the correction map as a function with position coordinates of the image as variables, and estimates the correction map based on distance information corresponding to the planar areas. [Effects of the Invention]

[0006] According to the present invention, distance measurement errors occurring across the entire screen can be reduced, and high-precision distance measurement becomes possible even if the device changes in position or temperature fluctuates. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment; [Figure 2] FIG. 1 is a diagram illustrating an optical system and an imaging unit according to a first embodiment. [Figure 3] FIG. 1 is a block diagram showing the configuration of an image processing unit according to a first embodiment; [Figure 4] FIG. 1 is a flowchart illustrating a correction map estimation process according to a first embodiment; [Figure 5] An example of an input image according to the first embodiment [Figure 6] Graph example when distance information is generated according to the first embodiment [Figure 7] Graph example showing correction amount of field curvature according to the first embodiment [Figure 8] FIG. 1 is a diagram illustrating an image used to estimate a correction map according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing the configuration of an imaging apparatus according to a second embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings, but the present invention is not limited to the contents described in each embodiment. The present invention allows for combinations of the configurations shown in each embodiment, as well as various modifications and variations. [Example]

[0009] In the first embodiment, a use case will be described in which an image capturing device having an image plane phase difference ranging function is used to capture an image of a person indoors. This embodiment can also be applied to an image processing device capable of acquiring an image captured using an external image capturing device and performing distance measurement calculations. The image processing device may include a mobile phone including a digital camera or a smartphone, a game console, a tablet terminal, a watch-type or eyeglass-type information terminal, a medical device, and a device for a monitoring system or an in-vehicle system.

[0010] 1 is a block diagram showing the configuration of an image processing device to which the present invention can be applied, when the image processing device is applied to an imaging device 100. The imaging device 100 includes an optical system 101, an imaging unit 102, an A / D conversion unit 103, a control unit 104, an image processing unit 105, a recording unit 106, a volatile memory 107, and a non-volatile memory 108.

[0011] The optical system 101 includes a lens group consisting of a zoom lens and a focus lens, an aperture adjustment device, and a shutter device. The optical system 101 adjusts the magnification, focus position, or light amount of the subject image that reaches the imaging unit 102.

[0012] The image capturing unit 102 is a photoelectric conversion element such as a CCD or CMOS sensor that photoelectrically converts the light beam of the subject that has passed through the optical system 101 into an electrical signal.

[0013] A drive control unit may be provided for each of the optical system 101 and the imaging unit 102. The drive control unit drives the optical system and the imaging unit in response to instructions from a control unit 104 (described later) and can automatically adjust the focus position.

[0014] The A / D converter 103 converts the input analog image signal into digital image signal data.

[0015] Furthermore, the imaging unit 102 may be an imaging surface phase difference sensor that includes an A / D conversion unit and outputs phase difference information or defocus information converted from the phase difference information to the control unit 104 described below.

[0016] The image processing unit 105 performs the parallax calculation process of the present invention in addition to normal signal processing on the data from the A / D conversion unit 103 etc. Here, the normal processed signal refers to noise reduction processing, development processing, and processing for compressing gradation to a predetermined output range by gradation compression processing using gamma conversion.

[0017] The control unit 104 is a control unit that is made up of at least one processor or circuit, and controls the entire image capture device 100. For example, it includes a processor such as a CPU or MPU, and controls the operation of each block of the image capture device 100 by loading a program recorded in a non-volatile memory 108 into a volatile memory 107 and executing the program. For example, it calculates the amount of exposure during shooting to obtain an input image with appropriate brightness, and controls the optical system 101 and image capture unit 102 to achieve this, thereby controlling the aperture, shutter speed, and analog gain of the sensor.

[0018] The recording unit 106 has a function of recording images, and may include, for example, an information recording medium using a memory card equipped with a semiconductor memory or a package containing a rotary recording medium such as a magneto-optical disk.

[0019] Fig. 2(A) shows the positional relationship between the optical system 101 and the imaging unit 102 in the imaging device 100 of Fig. 1. A dashed line 101a indicates the optical axis of the optical system 101, and the imaging unit 102 is disposed so as to be approximately perpendicular to the optical axis 101a.

[0020] 2(B) shows the pixel array configuration of the imaging unit 102 in FIG. 1. Here, the optical axis 101a coincides with the z-axis direction. As shown in the cross-sectional view of FIG. 2(C), each pixel 200 is composed of a microlens 201, a color filter 202, and photoelectric conversion units 203A and 203B. In the imaging unit 102, each pixel is given red, green, and blue spectral characteristics corresponding to the wavelength band to be detected by the color filter 202, and the color filters are arranged according to a known color pattern. Photoelectric conversion units 203A and 203B having sensitivities corresponding to the wavelength bands are formed on a substrate 204.

[0021] FIG. 2(D) is a view of the exit pupil viewed from the intersection of the optical axis 101a of the optical system 101 and the imaging unit 102. A light beam that has passed through a first pupil region 210, which is a different region of the exit pupil, and a light beam that has mainly passed through a second pupil region 220 are incident on the photoelectric conversion units 203A and 203B, respectively. By photoelectrically converting the light beams incident on the photoelectric conversion units 203A and 203B, images A and B that have been captured from different viewpoints are generated. The detected images A and B are transmitted to the image processing unit 105, where distance information is calculated by distance measurement calculation processing and stored in the recording unit 106. In addition, an image obtained by adding images A and B can be used as image information.

[0022] In FIG. 2(D), reference numeral 211 denotes the center of gravity of the pupil region 210, and reference numeral 221 denotes the center of gravity of the pupil region 220. In this embodiment, the center of gravity 211 of the pupil region 210 is decentered (moved) in the x-axis direction from the center of the exit pupil, and the center of gravity 221 of the pupil region 220 is decentered (moved) in the opposite direction. The distance between these centers of gravity is the base length 222, and the direction of this line segment is called the pupil division direction. The positions of image A and image B change in the same direction as the pupil division direction. The amount of relative position change between these images, i.e., the parallax between image A and image B, is a value that corresponds to the amount of defocus. This parallax can be detected and converted using known techniques and converted into distance information such as the amount of defocus and the distance from the imaging device to the subject.

[0023] Next, the configuration of the image processing unit 105 according to this embodiment will be described with reference to Fig. 3. Note that each block of the image processing unit 105 may be realized by a combination of software and hardware. Also, multiple functional blocks may be integrated, or one functional block may be separated.

[0024] The signal processing unit 301 performs the above-mentioned normal signal processing such as noise reduction processing, development processing, etc. The signal processing unit 301 may combine the signals of image A and image B and handle them as a single image signal.

[0025] The shooting information acquisition unit 302 acquires various information such as the shooting mode set by the user at the time of shooting, focal length, aperture value, exposure time, etc., from the non-volatile memory 108 or the volatile memory 107, for example, via the control unit 104, and provides it to the distance information generation unit 305.

[0026] The correction information acquisition unit 303 acquires various types of routine information that are not included in the shooting settings, such as correction information for field curvature and correction coefficients for distortion aberration, from the non-volatile memory 108 or the volatile memory 107, for example, via the control unit 104, and provides the information to the distance information generation unit 305.

[0027] The plane detection unit 304 detects a plane area of ​​the subject based on the image signal output from the signal processing unit 301 .

[0028] The distance information generating unit 305 acquires the image signals of the A image and the B image, and generates distance information such as the defocus amount and the distance from the imaging device to the subject based on the parallax, the shooting information, and the correction information.

[0029] The correction map estimation unit 306 outputs a correction map for correcting distortion in the distance information using the outputs of the plane detection unit 304 and the distance information generation unit 305. The correction map is stored in the nonvolatile memory 108 or the volatile memory 107 via the control unit 104, for example, and is used to generate the next distance information.

[0030] A series of operations in the correction map estimation process according to this embodiment will be described below with reference to Fig. 4 and Fig. 5. Fig. 5 is a diagram of an example of an input image 501. The input image 501 is an image captured of a scene in which three people 502, 503, and 504 are standing indoors.

[0031] In S401 , the distance information generation unit 305 acquires the A image and the B image of the input image 501 .

[0032] In S402, the distance information generation unit 305 generates distance information based on the image A and the image B. Specifically, the disparity between the image A and the image B is calculated, the disparity is converted into a defocus amount, and then the defocus amount is converted into a distance to the object surface using a lens formula. The calculation of the disparity and the conversion of the defocus amount may be performed using a known method such as that disclosed in Japanese Patent Application Laid-Open No. 2016-9062, to generate a distribution of the defocus amount for each pixel.

[0033] Here, since distance conversion is performed taking into account the amount of defocus caused by field curvature, the defocus amount is corrected using field curvature correction map W. If the defocus amount before correction is L and the defocus amount after correction is L', then L' = L + W is calculated. If the states of the optical system 101 and the imaging unit 102 do not change, the field curvature will not change. Therefore, it is possible to obtain the field curvature correction map W in advance using design data or measurement and record it in non-volatile memory 107, etc.

[0034] As described above, the distance from the imaging device to the subject can be calculated from the lens formula in geometric optics.

[0035]

number

[0036] In equation (1), the value of B is found from the distance from the principal point to the image sensor and the defocus amount, and F is calculated using design data or a measured value, allowing the distance A to be calculated for each pixel. Hereinafter, the distance from the image sensor to the subject calculated in this way will be referred to as the measured distance. The reference for the measured distance will be the principal point of the optical system if the value of A in equation (1) is used, but the reference for the measured distance can also be changed to the sensor, lens tip, etc. by providing an appropriate offset value.

[0037] In S403, the plane detection unit 304 detects plane areas from the subject captured in the input image 501 based on the image signal. An existing method may be used for the detection, and for example, image segmentation using a neural network may be used to identify areas estimated to be the walls, floor, and ceiling of a room as planes. If a plane is detected, the process proceeds to S404; if not, the correction map estimation process ends.

[0038] In S404, the correction map estimation unit 306 determines whether to estimate a correction map based on distance information corresponding to the plane detected by the plane detection unit 304 (hereinafter referred to as the plane).

[0039] FIG. 6 is a graph showing distance measurements on dashed line 510 in FIG. 5. Dashed line 510 spans indoor walls 506, 507, and 509, which are planar objects, and people 502, 503, and 504, which are non-planar objects. When the distance information generated by distance information generation unit 305 is correct, the amount of change in distance measurements in a planar area is constant. FIG. 6(A) is a graph when distance information is generated correctly, and the amount of change in distance measurements corresponding to the plane is constant. On the other hand, FIG. 6(B) shows a case where distance information is not generated correctly, and the distance information corresponding to the plane is a curve (curved surface). In this way, when the amount of change in distance measurements corresponding to the plane is not constant, correction map estimation unit 306 starts estimating a correction map.

[0040] A specific determination method may be, for example, to calculate the difference in distance measurement values ​​between adjacent pixels on a certain plane, and start correction when the amount of change in the difference between adjacent pixels exceeds a certain threshold. Alternatively, correction may be started when the amount of change within a planar region exceeds a certain threshold. Furthermore, distance measurement values ​​for each pixel on the same plane may be extracted, fitted to form a plane, and the obtained distance value may be compared with the distance measurement value of each pixel for determination. The error between the distance value on the plane of each pixel and the distance measurement value is calculated, and estimation of a correction map may be started when the average error exceeds a threshold. Alternatively, estimation of a correction map may be started when the variation in error within a certain region of each plane exceeds a specific threshold.

[0041] In S405, the correction map estimation unit 306 estimates a correction map based on distance information corresponding to the plane so that the distance measurement value approaches the plane. The correction map is expressed as a function with the position coordinates of the image as variables. That is, the correction map can be treated as having a correction value for each pixel. In this embodiment, the change amount ΔW in the field curvature correction map W is estimated as the correction map. Also, in this embodiment, ΔW is a quadric surface with the x and y coordinates of the image as variables, as in Equation (2). However, any function that is continuous and has a maximum (or minimum) at a certain position, such as a spherical or bowl-shaped function, may be used. Since ΔW is the change amount of the field curvature correction map W, the defocus amount L′ after correction is expressed as L′ = L + W + ΔW. a1x 2 +a2y 2 +a3z 2 +a4xy+a5yz+a6zx+a7=0···(2) a 1,2,···7 :parameters of quadratic surface (x,y): Image position coordinates z: Correction value

[0042] The process of estimating the correction map will now be explained. The defocus amount L' corresponding to the plane is extracted, and the parameters of the quadric surface are estimated based on this value. According to the lens formula, the distance measurement value A of each pixel taking into account ΔW is calculated using equation (3) (where B = L').

[0043]

number

[0044] The quadric surface parameters are estimated so that the distance measurement value A for each pixel calculated according to equation (3) approaches the plane as closely as possible in each plane region. A known method may be used to estimate the parameters. For example, as in the correction start determination process described above, an optimization problem can be solved to minimize evaluation values ​​such as the amount of change in the difference in distance measurement values ​​between adjacent pixels, the difference from the distance value calculated by fitting, and the variance in the difference. However, because distance measurement errors across the entire screen must be taken into consideration, it is necessary to calculate evaluation values ​​for all planes present in the image and minimize their sum.

[0045] Furthermore, in this embodiment, since the correction takes into account the curvature of field, the offset component of the quadric surface is determined based on the pre-correction distance measurement value A at the imaging section near the optical axis of the optical system 101, which is less susceptible to the influence of the curvature of field.

[0046] Since ΔW uses the position coordinates of the image as a variable, correction can be performed even in areas that are not flat.

[0047] The graph in Fig. 7 shows the amount of correction for field curvature on dashed line 510. Dotted line 701 is the original field curvature correction map W. Solid line 702 is W+ΔW in a planar region, and dashed line 703 is W+ΔW in a non-planar region. As described above, the correction map estimation unit 306 can estimate a correction map for the entire image based on the distance measurement values ​​of the plane.

[0048] In S406, the distance information generation unit 305 regenerates the distance information using the correction map estimated by the correction map estimation unit 306. As a result, corrected distance information is generated in which the distance information of the entire screen has been corrected.

[0049] The above process corrects the distance information for the entire screen. By estimating an appropriate correction map based on the distance measurement values ​​of the plane and generating corrected distance information, distortion components in the distance measurement values ​​are reduced, enabling highly accurate distance measurement.

[0050] If the field curvature changes due to temperature changes or changes in the posture of the imaging device, the factory-set field curvature correction map W cannot fully correct the change, resulting in a ranging error that appears as distortion across the entire screen. Because the field curvature can be expressed by a predetermined function that corresponds to the position coordinates of the image, it is possible to estimate a correction map for the entire screen using information on a partial flat area of ​​the screen.

[0051] In this embodiment, plane detection is performed automatically by image processing, but it is also possible to allow the user to directly specify planes using an imaging device equipped with a touch panel that displays images and has a touch input function. In other words, the user can specify as a detection target a plane that is difficult to determine by image processing, or specify a plane that has been erroneously detected to be excluded from the detection target, thereby enabling a correction map to be estimated with higher accuracy.

[0052] In this embodiment, the correction map is estimated using all distance information of the plane, but the correction map may be estimated based on more reliable distance information, such as a high-contrast area where parallax calculation is easy or an area with a high correlation value in parallax calculation, etc. By doing so, the correction map can be estimated with higher accuracy.

[0053] In this embodiment, the correction map is estimated using one pair of images A and B. However, two or more pairs of images A and B may be used to estimate the correction map based on the distance measurement values ​​of the planes in each pair. That is, image signals obtained by capturing images multiple times may be used. For example, in addition to image 501, image 801 as shown in FIG. 8A may also be used for estimation. Image 801 has a slightly different scene from image 501. The positions of people 502 and 503 have moved, person 504 has disappeared from the imaging range, and the orientation of the imaging device is also different from that in image 501. FIG. 8B shows image 501 superimposed on image 801 with dotted lines. Ceiling 508 and right wall 507 are planar regions in each image, and distance information calculated from each image can be used. Furthermore, vertical line region 802 has a planar region in a different position from image 501, and distance information from a wider planar region can be used compared to image 501 alone. In either case, the increased distance information for the planar region allows for more robust and accurate estimation of the correction map.

[0054] The image processing device may be provided with a notification function that displays on the screen areas where no planar distance information exists, making it easier for the user to capture new images required for more accurate correction.

[0055] In this embodiment, a correction map is generated when it is determined that the distance measurement value of the plane has not been generated correctly, but the correction map may be estimated based on other criteria. For example, a temperature sensor or an attitude sensor may be attached to the imaging device, and the correction map may be estimated when the amount of change in these sensors exceeds a certain threshold. Furthermore, the sensor values ​​and the correction map values ​​may be stored in the nonvolatile memory 108, and correction may be performed using a correction map estimated in the past according to the current sensor value.

[0056] Furthermore, the image processing device may be provided with an area detection means for detecting an area where correction by the correction map is estimated to be insufficient. In this case, providing an area notification means for notifying the detected area makes it easier for the user to capture a new image that requires more accurate correction. The image processing device may be provided with a function such as a switch that controls whether or not to estimate the correction map. By not performing unnecessary plane detection processing, the power consumption of the image processing device can be reduced.

[0057] Furthermore, the control unit 104 may drive the drive control unit based on distance information calculated by the image processing device. This allows the optical system and / or the imaging unit to be driven to automatically adjust to an appropriate focus position, allowing the user to easily take a photograph with the subject in focus. [Example]

[0058] In the second embodiment, an image processing device to which the present invention can be applied will be described using an image capturing device that captures stereo images as an example. Note that the description of the same configuration and processing as in the first embodiment will be omitted.

[0059] This embodiment can also be applied to an image processing device that acquires an image captured using an external imaging device and is capable of performing distance measurement calculations. Image processing devices may include digital cameras, mobile phones including smartphones, game consoles, tablet devices, watch-type and eyeglass-type information terminals, medical devices, and devices for monitoring systems and in-vehicle systems.

[0060] 9(a) is a block diagram showing the configuration of an image processing device to which the present invention can be applied, when the image processing device is applied to an imaging device 900. The imaging device 900 includes optical systems 9011 and 9012, imaging units 9021 and 9022, A / D conversion units 9031 and 9032, a control unit 904, an image processing unit 905, a recording unit 906, a volatile memory 907, and a non-volatile memory 908.

[0061] The components of the imaging device 900 have the same functions as the components of the first embodiment except for the optical systems 9011 and 9012 and the imaging units 9021 and 9022, and therefore a description thereof will be omitted.

[0062] The optical systems 9011 and 9012 are the same type of optical system equipped with a lens group consisting of a zoom lens and a focus lens, an aperture adjustment device, and a shutter device. These optical systems 9011 and 9012 adjust the magnification, focus position, or light amount of the subject image that reaches the imaging units 9021 and 9022, respectively, and these parameters are adjusted to be the same for each optical system.

[0063] The imaging units 9021 and 9022 are photoelectric conversion elements such as CCD or CMOS sensors that photoelectrically convert the light beams of the subject that have passed through the optical systems 9011 and 9012, respectively, into electrical signals.

[0064] A / D converters 9031 and 9032 convert analog image signals input from the image pickup units 9021 and 9022 into digital image signal data.

[0065] Furthermore, the image capturing units 9021 and 9022 may be image capturing surface phase difference sensors that include an A / D conversion unit and output defocus amount information to the control unit 904, which will be described later.

[0066] FIG. 9(b) shows the positional relationship between optical systems 9011 and 9012 and imaging units 9021 and 9022 of the imaging device 900. The optical systems 9011 and 9012 are arranged so that their optical axes 9011a and 9012a are approximately parallel. The distance between the principal points of the optical systems 9011 and 9012 is the base length 921. The imaging units 9021 and 9022 are arranged approximately on the same plane and generate images A and B, respectively. The relative positional change in the image in the base length direction of the same subject appearing in images A and B is called parallax. The distance from the imaging device to the subject can be calculated by detecting and converting this parallax using a known method.

[0067] A series of operations in the correction map estimation process according to this embodiment will be described below. However, since the processes of image acquisition, plane detection, and correction map estimation start are the same as those in Example 1, they will be omitted, and only the distance information generation and correction map estimation process will be described.

[0068] The distance information generation unit 305 of the image processing unit 905 generates distance information from the image capture device to the subject using a known method based on image A and image B. The process of generating distance information is divided into three stages: image preprocessing, parallax calculation, and distance conversion.

[0069] First, we will explain image preprocessing. Image A and image B each contain distortion, which, if left as is, would result in distance measurement errors depending on the image region. Therefore, the distance information generation unit 305 performs distortion correction using the distortion correction parameter k acquired from the correction information acquisition unit. Distortion generally occurs in the circumferential and tangential directions around the optical axis. Therefore, distortion can be modeled, for example, as shown in equation (4).

[0070]

number

[0071] The distortion correction parameter k does not change unless the state of the optical systems 8011 and 8012 changes, so basically the parameter k recorded in the nonvolatile memory 807 at the time of shipment from the factory is used.

[0072] Next, the distance information generating unit 305 calculates the disparity by using a known method, such as block matching or phase-only correlation.

[0073] Finally, the distance information generating unit 305 converts the parallax calculated for each pixel into distance information from the image capturing device to the subject according to equation (5).

[0074]

number

[0075] The distance information from the imaging device to the subject generated in this manner is called a distance measurement value.

[0076] The correction map estimation unit 306 starts the process of estimating a correction map when the amount of change in the distance measurement value corresponding to the plane is not constant. When the lens distortion coefficient changes due to temperature changes in the imaging device, the imaging position of the light beam changes depending on the region, resulting in a distance measurement error that appears distorted across the entire screen. In this embodiment, the amount of change in the imaging position due to distortion is estimated as the correction map. That is, the correction map is a model similar to equation (4), and correction is performed by calculating the amount of change Δk in the distortion correction parameter k for each coefficient. As in the first embodiment, the correction method is performed by minimizing evaluation values ​​such as the change in the amount of change in the distance measurement value used in the process of determining whether to start correction, and the error and variation from the plane calculated by fitting, taking into account all planes of the image.

[0077] The above-described process allows an appropriate correction map to be estimated based on the distance measurement values ​​of the plane, thereby reducing distortion components in the distance measurement values ​​and enabling highly accurate distance measurement.

[0078] When distortion changes due to changes in temperature or device orientation, the factory-set distortion correction parameter k cannot fully correct the distortion, resulting in ranging errors that cause the entire screen to appear distorted. Because distortion can be expressed as a predetermined function that corresponds to the position coordinates of the image, it is possible to estimate a correction map for the entire screen using information on a partial flat area of ​​the screen.

[0079] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0080] 100 Imaging device 101 Optical system 102 Imaging unit 103 A / D conversion section 104 Criminal Justice Department 105 Image processing section 106 Recording Unit 107 Volatile Memory 108 Non-volatile memory

Claims

1. a generating means for generating distance information using a plurality of image signals captured from different viewpoints; a detection means for detecting a planar area in the image signal; an estimation means for estimating a correction map for correcting the distance information; Area detection means; an area notification means for notifying the area detected by the area detection means, the estimation means estimates the correction map based on distance information corresponding to the planar region; the generating means generates the distance information using the correction map; The image processing apparatus is characterized in that the area detection means detects an area in the image signal where correction is estimated to be insufficient even if the correction map is used.

2. the generating means generates the distance information using a field curvature correction map that corrects aberrations due to field curvature; 2. The image processing apparatus according to claim 1, wherein the estimation means estimates the correction map for correcting an error in the distance information due to the field curvature correction map.

3. 3. The image processing apparatus according to claim 2, wherein the correction map is a function expressed by a quadric surface.

4. the generating means generates the distance information based on the image signal after correcting distortion; 4. The image processing apparatus according to claim 3, wherein the function is expressed based on a distortion model.

5. The image processing device according to any one of claims 1 to 4, characterized in that the estimation means calculates a plane based on distance information corresponding to the planar area, and estimates a correction map so that the difference between the distance value from the obtained plane and the distance information is small.

6. 6. The image processing device according to claim 1, wherein the estimation means estimates the correction map so that a change amount of distance information corresponding to the planar area is constant in the planar area.

7. 7. The image processing apparatus according to claim 1, wherein the generating means generates corrected distance information by correcting the distance information using the correction map.

8. The image processing device according to any one of claims 1 to 7, characterized in that the generating means acquires image signals captured multiple times from different viewpoints and generates multiple pieces of distance information based on signals of each pixel, and the estimating means estimates the correction map based on the multiple pieces of distance information.

9. 9. The image processing apparatus according to claim 8, wherein the estimation means estimates the correction map based on the plurality of pieces of distance information at the same position coordinates of the image signal.

10. 9. The image processing apparatus according to claim 8, wherein the image signal is an image signal obtained by capturing an image of the planar region a plurality of times so that the position of the planar region is different.

11. 11. The image processing apparatus according to claim 1, wherein the detecting means has a function of determining whether a plane is included in the image signal.

12. 12. The image processing apparatus according to claim 11, further comprising a notification unit that notifies a user of information urging the user to photograph a plane when the detection unit determines that the image signal does not contain a plane.

13. 13. The image processing device according to claim 1, wherein the estimation means estimates the correction map when there is a change in distance information corresponding to the planar area that is greater than a predetermined threshold value.

14. 14. The image processing apparatus according to claim 1, wherein the estimation means estimates the correction map based on distance information in the planar region and the reliability of the distance information.

15. 15. An imaging apparatus comprising: an image processing device according to claim 1; and imaging means for capturing an object image formed via an optical system.

16. 16. The imaging device according to claim 15, wherein the imaging means generates the plurality of images based on signals generated by a plurality of photoelectric conversion units, with light beams passing through different pupil regions of an optical system being incident on a plurality of photoelectric conversion units.

17. The imaging device according to claim 15 or 16, further comprising a change detection means for detecting a change in temperature or posture of the imaging device, and the generation means generates the distance information when the change detection means detects a change amount greater than a predetermined threshold.

18. 18. The imaging device according to claim 15, further comprising a drive control means for controlling the drive of both the optical system and the imaging unit, or either one of the optical system and the imaging unit, wherein the drive control means drives the optical system and the imaging unit using distance information obtained by the image processing device, and automatically adjusts the focus position.

19. a generating step of generating distance information using a plurality of image signals captured from different viewpoints; a detecting step of detecting a planar area in the image signal; an estimation step of estimating a correction map for correcting the distance information; a region detection step; a region notification step of notifying the region detected by the region detection step, In the estimation step, the correction map is estimated based on distance information corresponding to the planar region; In the generating step, the distance information is generated using the correction map; The image processing method is characterized in that the area detection step detects an area in the image signal where correction is estimated to be insufficient even when the correction map is used.

20. A program that causes a computer to execute each step of the control method according to claim 19.

Images