Imaging device, parallax displacement correction method, and parallax displacement correction program

Abstract

Provided is an imaging device that can detect a road surface position with high reliability and that can thereby detect parallax displacement with high accuracy. This imaging device is provided with: a first imaging unit for acquiring a first captured image; and a second imaging unit that is disposed at a position away by a fixed distance from the first imaging unit and that acquires a second captured image. A parallax calculation unit calculates parallaxes in various regions in the first captured image and in various regions in the second captured image. A region selection unit selects, from the first captured image or from the second captured image, regions having a threshold value of reliability or higher as the value of each of the parallaxes. A parallax displacement amount calculation unit makes a comparison between the values of the parallaxes in the respective regions selected by the region selection unit and ideal values of parallaxes to be acquired in the regions, and obtains parallax displacement amounts of the regions. A parallax displacement correction unit corrects parallax displacements in the first imaging unit and the second imaging unit using correction amounts determined in accordance with the parallax displacement amounts of the regions.

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device, a parallax displacement correction method, and a parallax displacement correction program. [Background technology]

[0002] Stereo cameras are known as devices for recognizing objects three-dimensionally. Stereo cameras use the differences in the images captured by multiple cameras placed at different positions to detect parallax between the multiple cameras based on trigonometry, and then use this parallax to detect the distance and position of an object from the camera. Stereo cameras are installed in vehicles, for example, and used as cameras that constitute driving assistance systems.

[0003] However, if the optical axis of a stereo camera shifts due to deterioration over time, accurate parallax calculation becomes impossible. This makes it difficult to accurately measure the distance from the camera to an object, increasing the possibility that the driving assistance system will not function properly. For this reason, various techniques have been proposed for calibrating the optical axis (in other words, correcting parallax shift) in order to accurately calculate parallax (see, for example, Patent Document 1). Patent Document 1 discloses a technique for correcting parallax shift by calculating the parallax between the position of the road surface near the vehicle (nearby road surface) and the position of the road surface far from the vehicle (far road surface position). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-44573 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the device disclosed in Patent Document 1 has limitations on the environment in which the parallax displacement can be corrected because the road surface position used to calculate the parallax displacement is preset. Furthermore, the road surface has a monotonous texture, which can lead to low reliability in the parallax calculation. This can be particularly true on unclear roads lacking white lines or various signs. Calculating the amount of parallax displacement using a disparity with low reliability can lead to a decrease in the accuracy of the parallax displacement correction, and ultimately to a decrease in the accuracy of distance measurement.

[0006] The present disclosure proposes an imaging device, a parallax displacement correction method, and a parallax displacement correction program that are capable of detecting parallax displacement with high accuracy. [Means for solving the problem]

[0007] An imaging device according to one aspect of the present disclosure is characterized in that it includes a first imaging unit that acquires a first captured image, a second imaging unit that is positioned at a certain distance from the first imaging unit and acquires a second captured image, a parallax calculation unit that calculates parallax for each pixel of the first captured image and the second captured image, a region selection unit that selects a region from the first captured image or the second captured image that has a reliability of the parallax value equal to or higher than a threshold, a parallax shift amount calculation unit that compares the parallax value in the region selected by the region selection unit with an ideal value of parallax that should be acquired in the region, and calculates the parallax shift amount for each region, and a parallax shift correction unit that corrects the parallax shift in the first imaging unit and the second imaging unit using a correction amount that is determined by statistically processing the parallax shift amount for each region. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging device, a parallax displacement correction method, and a parallax displacement correction program that are capable of detecting parallax displacement with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1A] 1 is a schematic diagram illustrating the overall configuration of a stereo camera 100 as an imaging device according to a first embodiment. [Figure 1B] 1B is a block diagram illustrating an example of the detailed configuration of the image processing device 2 of FIG. 1A. FIG. [Figure 2A] 10 is an explanatory diagram illustrating a method for calculating the parallax D in the parallax calculation unit 13. FIG. [Figure 2B] 10 is an explanatory diagram illustrating a method for calculating the parallax D in the parallax calculation unit 13. FIG. [Figure 2C] FIG. 2 is an explanatory diagram illustrating a method for selecting an area in the area selection unit 14. [Figure 3A] 10 is an explanatory diagram illustrating a method for calculating the parallax D in the parallax calculation unit 13. FIG. [Figure 3B] 10 is an explanatory diagram illustrating a method for calculating the parallax D in the parallax calculation unit 13. FIG. [Figure 4A] 10 is an explanatory diagram illustrating a method for calculating a parallax D in the parallax calculation unit 13 and a method for calculating a parallax shift amount ΔD in the parallax shift amount calculation unit 15. FIG. [Figure 4B] 10 is an explanatory diagram illustrating a method for calculating a parallax D in the parallax calculation unit 13 and a method for calculating a parallax shift amount ΔD in the parallax shift amount calculation unit 15. FIG. [Figure 5] 10 is a flowchart illustrating a specific example of a procedure for calculating a parallax displacement correction amount ΔC in the stereo camera 100 according to the first embodiment. [Figure 6A] 10 is an explanatory diagram illustrating the operation of the area selection unit 14 in the stereo camera 100 according to the second embodiment. FIG. [Figure 6B] 10 is a flowchart illustrating a specific example of a procedure for calculating a parallax displacement correction amount ΔC in the stereo camera 100 according to the second embodiment. [Figure 6C] 10 is an explanatory diagram illustrating the operation of the area selection unit 14 in the stereo camera 100 according to the second embodiment. FIG. [Figure 7] FIG. 10 is an explanatory diagram illustrating the operation of the third embodiment. [Figure 8] FIG. 10 is an explanatory diagram illustrating the operation of the third embodiment. [Figure 9] FIG. 10 is an explanatory diagram illustrating the operation of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show embodiments and implementation examples according to the principles of the present disclosure, but these are for understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0011] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to implement the present disclosure, it should be understood that other implementations and forms are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0012] [First embodiment] The overall configuration of a stereo camera 100 as an imaging device according to a first embodiment will be described with reference to the schematic diagram of Fig. 1A. The stereo camera 100 is an in-vehicle stereo camera device mounted on a vehicle such as an automobile, and includes a stereo camera main body 1, an image processing device 2, and an interface 5. The stereo camera main body 1, the image processing device 2, and the interface 5 are connected via a communication line such as a bus BU.

[0013] The stereo camera main body 1 is configured by arranging a plurality of cameras, for example, two cameras, a first imaging unit 11 (left camera) and a second imaging unit 12 (right camera), spaced apart by a predetermined base line length B. The first imaging unit 11 and the second imaging unit 12 each capture an image of the area in front of the vehicle from positions different by the base line length B to obtain the image. The image processing device 2 may include, for example, a CPU 3 as a computing device and a memory 4 as a storage device. The memory 4 may be a RAM, a ROM, a hard disk drive (HDD), or a combination thereof. The stereo camera 100 may be connected to an external device 200 via an interface 5.

[0014] The external device 200 may include various sensors that detect the vehicle's accelerator opening (i.e., throttle opening), brake operation amount (brake pedal operation amount), steering angle, vehicle speed, acceleration, temperature, humidity, etc., as well as an ECU (Electronic Control Unit) that controls various vehicle operations.

[0015] The memory 4 is a recording medium that stores various information and programs (such as a parallax displacement correction program) used for various processes in the image processing device 2. The CPU 3 performs predetermined calculations on signals received via an interface (I / F) 5 in accordance with the control programs stored in the memory 4, detecting three-dimensional objects and the road surface, and calculating the position, distance, direction, etc. of objects. The calculation results by the CPU 3 are output to the external device 200 via the interface 5, and are used to determine and control various vehicle operations such as acceleration, braking, and steering.

[0016] 1B, an example of the detailed configuration of the image processing device 2 will be described. As an example, the image processing device 2 is configured with a parallax calculation unit 13, an area selection unit 14, a parallax shift amount calculation unit 15, and a parallax shift correction unit 16, which are realized by a program. The image processing device 2 calculates a distance Z from the stereo camera 100 to an object based on the parallax D calculated by the parallax calculation unit 13 and corrected by the parallax shift correction unit 16, and supplies the calculation result to the external device 200.

[0017] The parallax calculation unit 13 calculates the parallax D based on the image captured by the first imaging unit 11 and the image captured by the second imaging unit 12. Specifically, the parallax calculation unit 13 sets, for example, the image captured by the first imaging unit 11 as a reference image and extracts feature points with grayscale variations in the reference image. Next, the parallax calculation unit 13 uses the other image captured by the second imaging unit 12 as a reference image and searches the reference image for positions of reference points that reflect the same subject as the feature points extracted in the reference image. For the search, template matching such as Sum of Absolute Difference (SAD) can be used. The difference between the position of the extracted feature points in the reference image and their positions in the reference image is calculated as the parallax D. The calculated parallax D is calculated for each region containing the extracted feature points and is temporarily stored in the memory 4 in association with the position of the region. That is, the parallax calculation unit 13 is configured to extract multiple feature points from a pair of reference and reference images and calculate the parallax D for each of multiple regions containing the multiple extracted feature points.

[0018] The region selection unit 14 selects a region where a highly reliable parallax D has been obtained from among a plurality of regions in the captured image where data on the parallax D has been obtained. The region where a highly reliable parallax D has been obtained is selected, and the parallax D in the selected region is used to calculate the parallax shift amount ΔD, so that it becomes possible to perform parallax shift correction with high precision. A method for determining a highly reliable parallax D will be described later. Note that the number of regions to be selected is not important.

[0019] The parallax shift amount calculation unit 15 calculates the parallax shift amount ΔD for each region using the values ​​of parallax D for the multiple regions selected by the region selection unit 14. The parallax shift amount ΔD is calculated for each region as the difference (D-Di) between the parallax D calculated for a certain region by the parallax calculation unit 13 and the ideal value Di of parallax that should be obtained for that region. Here, the "ideal value Di of parallax" is determined by the heights of the first imaging unit 11 and the second imaging unit 12 from the object and the angle of view of the camera that captures the region. A specific procedure for calculating the ideal value Di of parallax will be described later.

[0020] The parallax displacement correction unit 16 determines a parallax displacement correction value ΔC for the captured image from the plurality of parallax displacement amounts ΔD calculated for each of the plurality of selected regions.

[0021] Next, with reference to FIGS. 2A to 2C, a method for calculating the disparity D in the disparity calculation unit 13 and a method for selecting a region having a highly reliable disparity D in the region selection unit 14 will be described. As shown in FIG. 2A, for example, an image of a road including a white line WL is captured by the first imaging unit 11 and the second imaging unit 12, and a first captured image IM1 and a second captured image IM2 are obtained. Here, the first captured image IM1 captured by the first imaging unit 11 is used as a base image, and the second captured image IM2 captured by the second imaging unit 12 is used as a reference image. The disparity calculation unit 13 extracts a certain feature point 21 in the first captured image IM1 and extracts a reference point 21' having the same feature as the feature point 21 in the second captured image IM2. The disparity calculation unit 13 then performs SAD template matching between the feature point 21 and the reference point 21', and calculates a similarity SM as shown in the graph of FIG. 2B, for example. The horizontal axis of the graph in FIG. 2B indicates the horizontal coordinate of the second captured image IM2, and the vertical axis indicates the SAD value (similarity SM).

[0022] When the similarity SM is calculated for the feature point 21 in the first captured image IM1 (base image) at the position of the reference point 21' in the second captured image IM2 (reference image) where the same object as the object captured at the feature point 21 is captured, the similarity SM will be a small value. On the other hand, when the similarity SM is calculated for the feature point 21 in the first captured image IM1 (base image) at a position in the second captured image IM2 (reference image) where an object different from the object captured at the feature point 21 is captured (a position other than the reference point), the similarity SM will be a high value. Here, the horizontal coordinate value of the position (first peak P1) where the similarity SM is minimum is the disparity D of the feature point 21.

[0023] When the protrusion amount of the first peak P1 (the amount of decrease in the similarity SM at the first peak P1) is k, the larger the value of the protrusion amount k, the greater the feature amount (e.g., luminance difference) of the feature point 21 relative to the adjacent region. The greater the luminance difference relative to the adjacent region, the greater the calculated protrusion amount k, and the higher the reliability of the disparity D calculated in that region. Therefore, the average value Av of the similarity SM is calculated for the region excluding the vicinity of the first peak P1 (the region surrounded by the dashed line in the figure), and this is used as a reference value. The difference between this average value Av (reference value) and the reliability SM of the minimum value of the first peak P1 is calculated as the protrusion amount k. The protrusion amount k is calculated for multiple regions, and the region selection unit 14 selects multiple regions with high k values. For example, as shown in FIG. 2C, the protrusion amount k is calculated for each of multiple regions (each assigned a unique region ID), and the regions are sorted in descending order of the protrusion amount k. A predetermined number of regions are selected in descending order of the protrusion amount k by the region selection unit 14. As will be described later, a threshold value THk may be set for the protrusion amount k, and if the number of regions where the protrusion amount k exceeds the threshold value THk does not reach the minimum value Nreg, the operation of parallax displacement correction may be stopped.

[0024] In Figures 2A and 2B, the protrusion amount k is calculated from the difference between the average value Av of the similarity SM (excluding values ​​near the first peak P1) and the lower limit value of the first peak P1. However, as shown in Figures 3A and 3B, a second peak P2 other than the first peak P1 (for example, a peak with the next largest protrusion amount k after the first peak P1) may be identified, and the difference between the lower limit value of this first peak P1 and the lower limit value of the second peak may be calculated as the protrusion amount k.

[0025] 4A, a method for calculating the parallax D in the parallax calculation unit 13 and a method for calculating the parallax shift amount ΔD in the parallax shift amount calculation unit 15 will be described. If the focal length of the lens L of the first imaging unit 11 and the second imaging unit 12 is f (mm), the pixel pitch of the imaging elements (e.g., CMOS sensors) of the first imaging unit 11 and the second imaging unit 12 is δ (mm / pixel), the position where the road surface at a position horizontally distant from the lens L by a distance Z (mm) is projected onto the imaging surface is J (pixel), and the height of the lens L from the road surface is h (mm), then the relationship shown in the following equation (1) holds.

[0026] [Number 1] Z = (f × h) / (J × δ) … (1)

[0027] Furthermore, when the distance (baseline length) between the first imaging unit 11 and the second imaging unit 12 is B (mm), the parallax D (pixels) at the distance Z (mm) satisfies the relationship shown in the following formula (2).

[0028] [Number 2] D = (B × f) / (Z × δ) … (2)

[0029] Substituting equation (1) into equation (2) gives the following equation (3).

[0030] [Number 3] D = B × J / h … (3)

[0031] In this way, the parallax D (pixel) of the road surface can be calculated from the base line length B (mm) and height h of the stereo camera 100, and the coordinate J (pixel) of the road surface on the imaging surface. The base line length B and height h (mm) are values ​​that can be identified when the first imaging unit 11 and the second imaging unit 12 are mounted on a vehicle, and therefore the parallax D (mm) can be uniquely identified by the coordinate J (pixel) of the road surface.

[0032] The ideal value Di of the parallax D of the road surface can be calculated based on the value of the base line length B and the value of the height h when the stereo camera 100 is mounted on the vehicle. Here, instead of calculating the ideal value Di of the parallax at the road surface, it is also possible to calculate the ideal value Di of the parallax at an object Ob (e.g., a curb, a guardrail, etc.) whose height from the road surface is known, as shown in FIG. 4B. However, since the road surface is a feature that is always present in a general driving environment, it is preferable in principle to calculate the ideal value Di of the parallax at the road surface. While the ideal value Di of the parallax is calculated at the road surface in principle, it is also possible to calculate the ideal value Di of the parallax at an object other than the road surface in special cases.

[0033] The parallax shift amount calculation unit 15 calculates the difference between the parallax D actually calculated in the parallax calculation unit 13 according to the base line length B and height h at that time and the ideal value Di of the parallax calculated according to the base line length B and height h at the time of design, as the parallax shift amount ΔD=D−Di. The parallax shift amount calculation unit 15 calculates the parallax shift amount ΔD for each of the multiple road surface areas selected by the area selection unit 14.

[0034] Next, a specific example of the procedure for calculating the parallax displacement correction amount ΔC in the stereo camera 100 of the first embodiment will be described with reference to the flowchart of FIG.

[0035] First, the road surface that is the basis for calculating the parallax displacement correction amount ΔC is identified (step S101). Then, in the same manner as described above, the parallax calculation unit 13 calculates the parallax D on the road surface for each region that includes the feature point 21 (step S102), and calculates the value of the protrusion amount k for each region (step S103).

[0036] Once the feature amount k has been calculated for each region, the region selection unit 14 selects regions whose protrusion amount k is greater than a threshold value THk (step S104). If the number of regions Nreg selected in step S104 is equal to or greater than the threshold value THreg, the process proceeds to step S106, but if it is smaller than the threshold value THreg, the calculation of the parallax displacement correction amount ΔC and the parallax displacement correction process are stopped (step S105). This is because if a certain number of regions whose protrusion amount k is equal to or greater than the threshold value THk are not obtained, it will be impossible to calculate the parallax displacement amount with high accuracy.

[0037] In step S106, the parallax displacement calculation unit 15 calculates the ideal value Di of parallax in each of the selected regions in the above manner, and calculates the difference between this ideal value Di and the actual parallax D as the parallax displacement ΔD=D−Di (step S107). Then, the parallax displacement correction unit 16 statistically processes the parallax displacement ΔD obtained in this way for the multiple regions to calculate the parallax displacement ΔDfix for the entire image, and calculates the parallax displacement correction amount ΔC in accordance with this parallax displacement amount ΔDfix (step S108).

[0038] As described above, according to the stereo camera 100 of the first embodiment, an area where a high protrusion amount k is obtained is selected, and the parallax shift amount ΔD is calculated based on the selected area, so that it is possible to provide an imaging device that can detect parallax shift with high accuracy.

[0039] [Second embodiment] Next, a stereo camera 100 according to a second embodiment will be described with reference to Figures 6A to 6C. The overall configuration of the stereo camera 100 according to the second embodiment is the same as that of the first embodiment (Figures 1A and 1B), so a duplicated description will be omitted.

[0040] In the first embodiment described above, the region selection unit 14 calculates the prominence k of the similarity SM (see FIG. 2B ), and selects multiple regions to be used in calculating the parallax shift amount ΔD according to the magnitude of the prominence k. In contrast, in the second embodiment, instead of obtaining a graph of the similarity SM, the region selection unit 14 determines the magnitude of the luminance difference between the region of interest 41 and a corresponding region, for example, an adjacent region 42 adjacent to the region of interest 41, as shown in FIG. 6A , and selects a region having a highly reliable parallax based on the magnitude of the luminance difference.

[0041] 6B, a specific example of the procedure for calculating the parallax displacement correction amount ΔC in the stereo camera 100 of the second embodiment will be described. First, the road surface that serves as the basis for calculating the parallax displacement correction amount ΔC is identified (step S201). Then, the area selection unit 14 calculates the luminance difference between a target area 41 on the identified road surface and an adjacent area 42 adjacent to the target area 41 (step S202). After the luminance difference has been calculated for each area, the area selection unit 14 selects areas where the luminance difference is greater than a threshold value THin (step S203).

[0042] Next, it is determined whether the number Nreg of regions selected in step S203 is equal to or greater than a threshold value THreg' (step S205). If the determination is Yes, the process proceeds to step S206, but if the determination is No (smaller than the threshold value THreg), the calculation of the parallax displacement correction amount ΔC and the parallax displacement correction process are stopped.

[0043] In step S206, the ideal value Di of parallax in each of the selected multiple regions is calculated in the above manner, and the difference between this ideal value Di and the actual parallax D is calculated as the parallax shift amount ΔD=D−Di in the parallax shift amount calculation unit 15 (step S207). Then, the parallax shift amount ΔD obtained in this way for the multiple regions is statistically processed to calculate the parallax shift amount ΔDfix for the entire image, and the parallax shift correction amount ΔC is calculated in accordance with this parallax shift amount ΔDfix (step S208).

[0044] 6C, it is also possible to calculate the luminance difference between the region of interest 41 and a nearby region 42' that is not adjacent to the region of interest 41 but is slightly away from it. The nearby region 42 is a region that is located at no distance from the region of interest 41, while the nearby region 42' is a region that is separated from the region of interest 41 by, for example, one region. If the edge portion of the region of interest 41 (for example, the edge portion between the road surface and the white line) becomes blurred due to the influence of the camera resolution, it may become impossible to correctly calculate the luminance difference between the region of interest 41 and the nearby region 42. However, by using the nearby region 42', which is slightly away, as the subject of the luminance difference calculation, it becomes possible to calculate the luminance difference more accurately.

[0045] As described above, according to the stereo camera 100 of the second embodiment, an area to be used for calculating the amount of parallax shift is selected in accordance with the magnitude of the luminance difference, and the amount of parallax shift ΔD is calculated based on the selected area. Therefore, similar to the first embodiment, it is possible to provide an imaging device that can detect parallax shift with high accuracy.

[0046] [Third embodiment] Next, a stereo camera 100 according to a third embodiment will be described with reference to FIGS. 7 and 8. The overall configuration of the stereo camera 100 according to the third embodiment is the same as that of the first embodiment (FIGS. 1A and 1B), so a duplicated description will be omitted. The third embodiment differs from the previous embodiments in the method of calculating the parallax shift amount in the parallax shift correction unit 16. Specifically, after region selection is performed by the region selection unit 14 as in the previous embodiments, a representative value (for example, median(J)) of the parallax shift amount ΔD of a plurality of regions for each ordinate J (1, 2, ... m, ...) is calculated in the parallax shift correction unit 16, as shown in FIG. 7.

[0047] Next, the average value of the calculated medians Median(1) to (m) is calculated as the average value ΔDav of the parallax displacement amount ΔD. This parallax displacement amount ΔDav is used as an input to calculate the parallax displacement correction amount ΔC in the parallax displacement correction unit 16. However, if the standard deviation SD of the median(J) used in calculating the average value ΔDav is larger than a threshold value, it is preferable not to perform parallax displacement correction on the captured image. This is because if the standard deviation SD is larger than a threshold value, it is determined that the reliability of the area containing that coordinate is low.

[0048] In the example shown in FIG. 8, one median (Median(j)) is calculated for one coordinate J, but one median (Median(k to k+x)) may be calculated collectively for multiple coordinates k to k+x. In particular, for road surfaces far from the vehicle (road surfaces shown in the upper part of the screen), it is preferable to calculate one median for multiple coordinates in the vertical direction. This is because road surfaces far from the vehicle are captured by the first and second imaging units 11 and 12 with a compressed amount of information per pixel compared to nearby road surfaces (road surfaces shown in the lower part of the screen). This leads to a decrease in the accuracy of parallax displacement correction. As described above, for distant road surfaces, calculating a median of 1 for multiple coordinates can prevent a decrease in accuracy.

[0049] Furthermore, as shown in FIG. 9, it is also possible to calculate the median of the parallax shift amount ΔD for all selected areas, rather than for each coordinate, and determine this median as the parallax shift amount ΔD for the captured image, and use this as the parallax shift correction amount ΔC.

[0050] Although various embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0051] 100...stereo camera, 200...external device, 1...stereo camera main body, 2...image processing device, 3...CPU, 4...memory, 5...interface, BU...bus, 11...first imaging unit, 12...second imaging unit, 13...parallax calculation unit, 14...area selection unit, 15...parallax shift amount calculation unit, 16...parallax shift correction unit.

Claims

1. a first imaging unit that acquires a first captured image; a second imaging unit disposed at a position spaced a predetermined distance from the first imaging unit and configured to acquire a second captured image; a parallax calculation unit that calculates parallax in each region of the first captured image and the second captured image; an area selection unit that selects an area from the first captured image or the second captured image, the area having a reliability of the disparity value equal to or greater than a threshold; a parallax shift amount calculation unit that compares the parallax value in the region selected by the region selection unit with an ideal value of parallax to be obtained in the region, and calculates the parallax shift amount for each region; a parallax displacement correction unit that corrects the parallax displacement in the first imaging unit and the second imaging unit using a correction amount determined according to the parallax displacement amount of each of the regions; Equipped with When the number of regions selected by the region selection unit is less than a predetermined number, the parallax displacement correction unit stops the parallax displacement correction. An imaging device characterized by:

2. the parallax calculation unit calculates parallax by performing template matching using the first captured image as a base image and the second captured image as a reference image; The imaging device according to claim 1 , wherein the region selection unit selects the region based on a prominence that is a difference between a minimum value of a peak in a graph of the similarity obtained in the template matching and a reference value.

3. The imaging device according to claim 2 , wherein the reference value is an average value of the similarity in a region excluding a region near the peak.

4. The imaging device according to claim 2 , wherein the reference value is a minimum value of a peak other than the peak.

5. The imaging device according to claim 1 , wherein the parallax displacement correction unit calculates a representative value of the parallax displacement amounts calculated for each region, and calculates a parallax displacement correction amount in accordance with the representative value.

6. The imaging device according to claim 1 , wherein the region selection unit selects the region based on a magnitude of a luminance difference between a region of interest in the first captured image or the second captured image and a region corresponding to the region of interest.

7. acquiring a first captured image captured by a first imaging unit and a second captured image captured by a second imaging unit disposed at a predetermined distance from the first imaging unit; calculating a parallax between each region of the first captured image and the second captured image; selecting an area from the first captured image or the second captured image, the area having a reliability of the disparity value equal to or greater than a threshold value; a step of comparing the parallax value in the selected region with an ideal value of parallax to be obtained in the region, and determining the parallax displacement amount for each region; correcting the parallax displacement in the first imaging unit and the second imaging unit by a correction amount determined in accordance with the parallax displacement amount of each of the regions; Including, When the number of regions selected in the region selection step is less than a predetermined number, the parallax displacement correction is stopped. Parallax correction method.

8. the step of calculating the parallax calculates the parallax by performing template matching using the first captured image as a base image and the second captured image as a reference image, 8. The parallax displacement correction method according to claim 7, wherein the step of selecting the region selects the region based on a prominence, which is a difference between a minimum value of a peak in a graph of the similarity obtained in the template matching and a reference value.

9. 9. The parallax displacement correcting method according to claim 8, wherein the reference value is an average value of the similarity in a region excluding a region near the peak.

10. The parallax displacement correcting method according to claim 8 , wherein the reference value is a minimum value of a peak other than the peak.

11. 8. The parallax displacement correcting method according to claim 7, wherein the correction of the parallax displacement comprises calculating a representative value of the parallax displacement amounts calculated in each region, and calculating a parallax displacement correction amount in accordance with the representative value.

12. The parallax displacement correction method according to claim 7 , wherein the step of selecting the region selects the region based on a magnitude of a luminance difference between a region of interest in the first captured image or the second captured image and a region corresponding to the region of interest.

13. acquiring a first captured image captured by a first imaging unit and a second captured image captured by a second imaging unit disposed at a predetermined distance from the first imaging unit; calculating a parallax between each region of the first captured image and the second captured image; selecting an area from the first captured image or the second captured image, the area having a reliability of the disparity value equal to or greater than a threshold value; a step of comparing the parallax value in the selected region with an ideal value of parallax to be obtained in the region, and determining the parallax displacement amount for each region; correcting the parallax displacement in the first imaging unit and the second imaging unit by a correction amount determined in accordance with the parallax displacement amount of each of the regions; configured to cause a computer to execute When the number of regions selected in the region selection step is less than a predetermined number, the parallax displacement correction is stopped. Parallax correction program.

Images