Stereo-image processor and method for correcting stereo-image

Abstract

To provide a stereo-image processor and a method for correcting a stereo-image which can correct a wide-angle camera equipped in a vehicle with high precision, without increasing the size of a chart.SOLUTION: A correction chart includes a pattern. The pattern has different lengths of baseline directions of a plurality of cameras in at least a first region and a second region. The correction processing unit determines a correction parameter on the basis of the result of conducting a stereo-matching, assuming that different patterns correspond to the same point, in the first region and in the second region of the correction chart.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to, for example, an in-vehicle stereo image processing apparatus and a stereo image calibration method mounted on a vehicle.

Background Art

[0002] As an apparatus for three-dimensionally recognizing an object, a stereo camera is known. A stereo camera utilizes the difference in the way images of a plurality of cameras arranged at different positions are captured, detects the parallax between the plurality of cameras based on triangulation, and uses the parallax to detect the depth and position of an object, and can accurately detect the position of a measurement target.

[0003] Such a stereo camera is mounted on a vehicle such as an automobile and applied to a technology (in-vehicle sensing technology) for detecting the position of an obstacle or the like. In in-vehicle sensing technology, in order to support many use cases, it is required to detect an obstacle or the like with a wide angle of view and to detect even a more distant obstacle (wide-angle conversion and far-field conversion).

[0004] In-vehicle stereo cameras are generally mounted inside an automobile in order to avoid the influence of dirt and the like, but with the wide-angle conversion, the influence of the front glass mounted on the vehicle cannot be ignored. Conventionally, in order to calibrate the misalignment of the stereo camera, a calibration operation of the stereo camera called aiming is performed during vehicle manufacture or inspection, but further measures are required for wide-angle conversion.

[0005] As a method for calibrating a stereo camera considering the influence of a windshield, Patent Document 1 is known. Patent Document 1 poses, as a problem, "to provide a calibration device and a calibration method for a stereo camera that can appropriately calibrate a stereo camera using a target board even in a narrow space." As a solution, it states, "A calibration device for a stereo camera that measures the distance to an object based on parallax includes a stereo camera 3 having a pair of cameras 4 and 6 arranged at a predetermined baseline length B, a target board 8 including at least two targets 8a and 8b arranged in parallel with the arrangement direction of the pair of cameras 4 and 6 and separated by the same distance as the baseline length B, and an arithmetic unit that calibrates the parallax shift of the pair of cameras 4 and 6 by treating the two targets 8a and 8b included in the captured images of the target board 8 captured by each of the pair of cameras 4 and 6 as the same target assuming that the target board 8 exists at an infinite distance from the stereo camera 3." Also, Patent Document 2 describes a similar calibration technique. Patent Document 2 poses, as a problem, "to variably set the search range when performing stereo matching according to the position on the image." As a solution, it states, "It has a comparison image line memory 7, an address generation circuit 10, and a stereo matching circuit 8. The line memory 7 stores the image data in the reference pixel region in one captured image and the image data on the horizontal line corresponding to the vertical position of the reference pixel region in the other captured image. The address generation circuit 10 sets the search range when performing stereo matching and instructs the line memory 7 to read the image data within the set search range and the image data within the reference image region. Also, the stereo matching circuit 8 identifies the correlation destination of the reference pixel region by stereo matching based on the image data within the search range read from the line memory 7 and the image data of the reference pixel region. Here, the above address generation circuit 10 calibrates the position of the search range regarding the reference pixel region based on the degree of deviation of the infinitely distant corresponding point with respect to the horizontal position of the reference pixel region."

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-62150 [Patent Document 2] Japanese Patent Application Laid-Open No. 2001-92968 [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] In Patent Document 1 and Patent Document 2, a chart is arranged near a stereo camera, the interval (baseline length) between the two cameras of the stereo camera is made to match the period of the pattern of the chart, and by assuming that the chart is virtually at infinity, it is stated that the influence of the inclination deviation of the light rays due to the refraction of the windshield can be calibrated. However, these techniques do not consider the influence of the position deviation of the light rays due to the refraction of the windshield, and there is a problem that it cannot be completely calibrated when a chart is actually arranged near the stereo camera.

[0008] The present invention has been made in view of the above problems, and provides a stereo image processing apparatus and a stereo image calibration method capable of calibrating a wide-angle stereo camera with high accuracy without increasing the size of the chart by arranging the chart near the stereo camera. [Means for Solving the Problems]

[0009] One aspect of the stereo image processing apparatus according to the present invention includes a stereo matching unit that performs stereo matching on a plurality of images captured by a plurality of cameras that image a subject through a light transmission member to obtain a disparity, and the stereo matching unit performs stereo matching on a plurality of images obtained by imaging a calibration chart with the plurality of cameras. A calibration processing unit that obtains calibration parameters for calibrating at least a disparity shift caused by the light transmission member based on the result of the stereo matching, the calibration chart includes a pattern, and the pattern is a repeating pattern larger than the baseline length of the plurality of cameras, and the calibration processing unit obtains the calibration parameters based on the result of performing stereo matching assuming that different patterns are at the same point.

Advantages of the Invention

[0010] According to the stereo image processing apparatus and the stereo image calibration method according to the present invention, it is possible to provide a stereo image processing apparatus and a stereo image calibration method that can calibrate a wide-angle camera with high accuracy without increasing the size of the chart.

[0011] Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0013] Hereinafter, this embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may sometimes be denoted by the same reference numerals. Although the accompanying drawings show embodiments in accordance with the principles of the present disclosure, these are for the purpose of understanding the present disclosure and are by no means used for interpreting the present disclosure in a limiting sense. The description in this specification is merely a typical example and does not limit the scope of the claims or the application examples of the present disclosure in any sense.

[0014] In this embodiment, although the description is made in sufficient detail for those skilled in the art to implement the present disclosure, other forms are also possible, and it is necessary to understand that changes in configuration and structure and replacement of various elements can be made without departing from the scope and spirit of the technical idea of the present disclosure. Therefore, the following description should not be construed as being limited thereto.

[0015] [Example 1] Referring to FIG. 1, the configuration of the stereo camera image processing apparatus 10 according to the first embodiment (hereinafter referred to as the "image processing apparatus 10" or the "stereo image processing apparatus 10") will be described. This image processing apparatus 10 is mounted on a vehicle such as an automobile, for example, and is used to detect the distance from the vehicle to three-dimensional objects (other automobiles, buildings, pedestrians, etc.) around the vehicle. Hereinafter, the case where the image processing apparatus 10 is mounted on a vehicle will be described as an example, but the present invention is not limited to this.

[0016] FIG. 1 is a block diagram showing a configuration example of the image processing apparatus 10 according to the first embodiment. This image processing apparatus 10 is configured to detect surrounding three-dimensional objects based on images obtained by the right camera 50 and the left camera 60, and issue an alarm as necessary. The right camera 50 and the left camera 60 constitute a stereo camera. Note that the cameras constituting the stereo camera are not limited to two on the left and right.

[0017] As an example, this image processing apparatus 10 includes an image processing unit 100, a stereo disparity image generation unit 200, a stereo disparity image calibration unit 99, a road surface cross-sectional shape estimation unit 400, a stereo three-dimensional object detection unit 500, and an alarm control unit 700.

[0018] In an area that can be commonly imaged by the right camera 50 and the left camera 60 (hereinafter referred to as the "stereo vision area"), the stereo disparity image generation unit 200 generates a stereo disparity image using the disparity between the right camera 50 and the left camera 60. Then, in the stereo three-dimensional object detection unit 500, the distance from the vehicle to the three-dimensional object is measured according to the disparity.

[0019] The right camera 50 and the left camera 60, although not shown in the figure, include a lens and an image sensor. The right camera 50 and the left camera 60 each acquire (image) an image of an object with the image sensor via the lens. In this embodiment, the right camera 50 and the left camera 60 are mounted inside the vehicle, and acquire an image of a subject around the vehicle via a front glass or the like which is a light transmissive member. The image processing apparatus 10 acquires an image P1 (first image) from the right camera 50 and an image P2 (second image) from the left camera 60.

[0020] The image processing unit 100 is configured to include, as an example, affine processing means 20a, 20b, luminance correction means 21a, 21b, pixel interpolation means 22a, 22b, and luminance information generation means 23a, 23b. The image processing unit 100 applies predetermined image processing to the images P1 and P2 obtained by the right camera 50 and the left camera 60, and supplies them to the stereo parallax image generation unit 200.

[0021] The affine processing means 20a applies affine processing to the image P1 from the right camera 50. The affine processing is, for example, a linear coordinate transformation process, but may include a non-linear operation. As a result of performing this affine processing, the affine processing means 20a acquires an image P3 (third image). Similarly, the affine processing means 20b applies affine processing to the image P2 from the left camera 60 to acquire an image P4 (fourth image).

[0022] The affine processing means 20a and 20b may also perform a distortion conversion process other than the affine processing. In this embodiment, the projection method fsinθ of the fisheye lens is projected and converted into the coordinate system of (ftanθx, ftanθy). Here, f is the focal length of the fisheye lens, θ is the angle of view incident on the fisheye lens, and θx, θy indicate the horizontal and vertical components of the angle of view incident on the fisheye lens. Further, in this embodiment, the displacement of the pixels in the vertical direction due to the influence of the front glass is corrected by the affine processing means 20a and 20b.

[0023] The luminance correction means 21a corrects the luminance of each pixel of the image P3. For example, the luminance of each pixel of the image P3 is corrected based on the gain of the right camera 50, the difference in gain of each pixel in the image P3, and the like. Similarly, the luminance correction means 21b corrects the luminance of each pixel of the image P4.

[0024] The pixel interpolation means 22a performs demosaicking processing on the image P3. For example, conversion from a RAW image to a color image is performed. Similarly, the pixel interpolation means 22b performs demosaicking processing on the image P4.

[0025] The luminance information generation means 23a generates luminance information of the image P3. For example, information representing a color image is converted into luminance information for generating a disparity image. Similarly, the luminance information generation means 23b generates luminance information of the image P4.

[0026] The stereo disparity image generation unit 200 generates a stereo disparity image of the stereo viewing region using the images of the aforementioned stereo viewing region (common viewing region) among the obtained images P3 and P4.

[0027] The stereo disparity image generation unit 200 includes an exposure adjustment unit 210 and a sensitivity correction unit 220, and can perform feedback control on the right camera 50 and the left camera 60 regarding the exposure amount, sensitivity, etc. of the right camera 50 and the left camera 60. Further, the stereo disparity image generation unit 200 further includes a geometric correction unit 230 that performs geometric correction on the left and right images, and a matching unit 240 that performs matching processing on the left and right images. The matching unit 240 performs stereo matching on the left and right images to obtain a disparity (stereo disparity image).

[0028] In the stereo disparity image calibration unit 99, based on the data of the stereo disparity calibration data recording unit 98 acquired in advance, the disparity shift due to a front glass or the like, which is a light transmission member, is calibrated.

[0029] The road surface cross-sectional shape estimation unit 400 estimates the cross-sectional shape of the road surface of the road on which the vehicle equipped with the image processing device 10 is scheduled to travel.

[0030] Based on the stereo disparity image generated by the stereo disparity image generation unit 200, the stereo vision three-dimensional object detection unit 500 detects three-dimensional objects in the stereo vision area. Also, for the detected three-dimensional objects, stereo matching is applied to detect the disparity, and the type of the three-dimensional object (such as pedestrian, bicycle, vehicle, building, etc.) is identified. By detecting three-dimensional objects and identifying their types, the types for further preventive safety use are specified. When a vehicle is detected, the detection result can be used for following control of the preceding vehicle, braking control in an emergency, etc. When the detected three-dimensional object is a pedestrian or a bicycle, emergency braking control and alarm control can be executed. Compared with stationary objects, for an object jumping out towards the vehicle, alarms and controls are to be implemented for objects within a wide field of view. By measuring the distance to these detected objects and estimating the moving speed of the objects being tracked in time series, more appropriate alarms and controls can be implemented by the alarm control unit 700.

[0031] In FIG. 1, in this embodiment, it is characterized by creating the data of the stereo disparity calibration data recording unit 98.

[0032] Next, the effects of this embodiment will be described. First, the problems of the prior art will be described in detail. As described above, in Patent Document 1 (hereinafter referred to as "Prior Art 1") and Patent Document 2 (hereinafter referred to as "Prior Art 2"), a chart is arranged near the stereo camera, the interval (baseline length) between the two cameras of the stereo camera is made to match the period of the chart pattern, and by assuming that the chart is virtually at infinity, it is said that the influence of the inclination deviation of the light rays due to the refraction of the windshield can be calibrated. On the other hand, there are mainly two influences due to the refraction of the windshield. The first is the image distortion due to the inclination deviation of the light rays. This image distortion occurs similarly for both the chart arranged nearby and the chart arranged far away. The second is the position deviation of the light rays. The position deviation of the light rays has a large influence on the chart arranged nearby and the measurement object, and has the characteristic that the chart arranged far away and the measurement object can be ignored.

[0033] Among these two factors, in Prior Art 1 and Prior Art 2, the influence of the displacement of the light rays due to the refraction of the windshield is not considered. As described above, this displacement of the light rays is detected when measuring a nearby object to be measured, but can be ignored when detecting a distant object to be measured. Usually, the parallax shift is calibrated so that the best distance measurement for distant objects is achieved. On the other hand, when calibrating the parallax shift using a nearby chart, a parallax shift occurs by the amount of displacement of the light rays due to the refraction of the windshield when detecting a distant object to be measured. For this reason, in Prior Art 1 and Prior Art 2, when a chart is placed nearby, there is a problem that the parallax shift cannot be completely calibrated.

[0034] Figure 2 is a diagram for explaining that problem. This figure shows a cross-section connecting the lens pupil positions of the right camera 50 and the left camera 60. Here, for simplicity of explanation, the amount of displacement of the light rays by the windshield 1 is shown larger than the actual amount. In this figure, the light rays at the horizontal viewing angle of 0 degrees in the right camera 50 and the left camera 60 are defined as light ray R51 and light ray R61, respectively. And the interval between the light ray R61 incident on the windshield 1 and the light ray R51 (the light ray R61 and the light ray R51 opposite to the stereo camera image processing device 10 with respect to the windshield 1) is defined as interval D1. Also, the light rays on the horizontal wide-angle side in the right camera 50 and the left camera 60 are defined as light ray R52 and light ray R62. Similarly, the interval between the light ray R62 incident on the windshield 1 and the light ray R52 is defined as interval D2. Also, the line obtained by extending the light ray R51 between the windshield 1 and the right camera 50 is defined as axis K51, and the line obtained by extending the light ray R61 between the windshield 1 and the left camera 60 is defined as axis K61. Similarly, the line obtained by extending the light ray R52 between the windshield 1 and the right camera 50 is defined as axis K52, and the line obtained by extending the light ray R62 between the windshield 1 and the left camera 60 is defined as axis K62. Here, the interval between axis K51 and axis K61 and the interval between axis K52 and axis K62 coincide with the baseline length B.

[0035] First, consider the horizontal viewing angle of 0 degrees. Although the ray R51 and the axis K51 are almost the same, they do not completely coincide. The same is true for the ray R61 and the axis K61. Therefore, the baseline length B and the distance D1 do not match. On the other hand, the distance D2 at a wide horizontal angle also differs from the baseline length B. Here, the characteristic point is that due to the influence of the front glass 1, the distances D1 and D2 are different. In the prior art 1 and the prior art 2, it is assumed that the influence of the distortion caused by the front glass is estimated by utilizing the fact that the distances D1, D2, and the baseline length B coincide. However, in reality, the distances D1, D2, and the baseline length B do not match.

[0036] FIG. 3 is a diagram for explaining the influence caused by the difference between the baseline length B and the distances D (distance D1, distance D2). Similar to FIG. 2, this figure shows a cross-section connecting the lens pupil positions of the right camera 50 and the left camera 60. Here, for simplicity of explanation, the amount of ray displacement caused by the front glass 1 is shown larger than the actual value. In the charts described in the prior art 1 and the prior art 2, it is assumed that a pattern with the same period as the baseline length is used. In this figure as well, a chart G10 with the same pattern period as the baseline length is arranged. In this figure, the axes of a predetermined viewing angle from the pupil positions of the right camera 50 and the left camera 60 are taken as the axes K53 and K63. Then, the contact points between the chart G10 and the axes K53 and K63 are respectively taken as the positions Q1 and Q2. Since the inclinations of the axes K53 and K63 are equal, the distance between the positions Q1 and Q2 is the baseline length B. This configuration is the same as that of the prior art 1 and the prior art 2. Then, the rays for detecting the images at the positions Q1 and Q2 by the right camera 50 and the left camera 60 are taken as the ray R53 and the ray R63. Then, the lines obtained by extending the rays R53 and R63 between the pupil positions of the right camera 50 and the left camera 60 and the front glass 1 are respectively taken as the line S53 and the line S63, and the intersection point of the line S53 and the line S63 is taken as the position TP1.

[0037] In the prior art 1 and the prior art 2, it is stated that by making the chart G10 into a pattern with the same period as the baseline length, virtual infinity can be detected. For this purpose, two axes such as the axis K53 and the axis K63 must be parallel lines. However, the angles of the light ray R53 incident on the right camera 50 and the light ray R63 incident on the left camera 60 through the front glass 1 are different. Therefore, the stereo camera adjusted with this chart generates a large error. In this figure, what should be infinity is detected assuming that the chart is at the position TP1, and being calibrated at the distance L becomes a major problem.

[0038] Therefore, this is taken into consideration in this embodiment. FIG. 4 shows the shape of the area of the chart G20 of this embodiment. Note that in each area, there is a pattern of the same pattern (a square shape in the illustrated example). In this embodiment, each area (pattern) of the chart G20 is characterized in that the width of the area in the horizontal direction (baseline direction) is different. For example, it shows that the horizontal width da1 of the area C33 (the first area) of the chart G20 and the horizontal width da2 of the area C35 (the second area) are different. The width da2 is larger than the width da1. And in this figure, when it is assumed that the plane including the lens optical axes of the two cameras of the stereo camera and the vertical bisector of the two cameras are on the area C33 on the chart G20 (in other words, the area including the position on the chart G20 where the plane including the lens optical axes of the two cameras of the stereo camera and the vertical bisector of the two cameras intersect is defined as the area C33, or the area on the chart G20 where the plane including the lens optical axes of the two cameras of the stereo camera and the vertical bisector of the two cameras intersect is defined as the area 33, and the area whose horizontal (baseline) position is different from that of the area C33 is defined as the area 35), both the width da1 and the width da2 are larger than the baseline length B, and the width da1 is closer to the baseline length B than the width da2 (that is, the width da2 is larger than the width da1). Note that as the curvature radius of the front glass becomes smaller, the width da1 and the width da2 move away from the baseline length B (see FIG. 11). On the contrary, as the curvature radius of the front glass becomes larger, the width da1 and the width da2 approach the baseline length B (see FIG. 11). Since the curvature radius of the front glass varies depending on the vehicle, it is necessary to set the chart interval accordingly. Also, when the distance between the stereo camera and the chart is small, the width da1 and the width da2 move away from the baseline length B. On the contrary, as the distance between the stereo camera and the chart increases, the width da1 and the width da2 approach the baseline length B.

[0039] FIG. 5 shows the patterns within each region of the chart G20 in FIG. 4. For example, it shows the pattern within region C33 on chart G20. Here too, the widths of the horizontal regions are different. For example, the horizontal width db1 at (0,0) and the horizontal width db2 at (2,0) are different. Width db2 is larger than width db1. And the detected light amount for each region is set to be different so that incorrect matching does not occur. With such a configuration, for example, the deviation amount of corresponding points between the image of region C34 of chart G20 detected by the right camera 50 and the image of region C33 detected by the left camera 60 is detected. As a method for obtaining the deviation amount, for example, detection methods such as block matching or feature point extraction used in a stereo camera may be used.

[0040] FIG. 6 shows the optical path when considering the positional deviation of light rays by the windshield. First, the light rays of the left camera 60 are the same as in FIG. 3. On the other hand, the angle of the light ray R63 incident on the left camera 60 and the angle of the light ray R54 incident on the right camera 50 are made to coincide (the lines S54 and S63 are parallel and the distance between them is the baseline length B), and the angle of the light ray R54 is changed. At this time, the horizontal periodic pattern on the chart G20 (in other words, the interval between the positions Q3 and Q2 of the images on the chart G20 detected by the right camera 50 and the left camera 60 with the light rays R54 and R63) is arranged at a distance D apart.

[0041] The parallax shift due to the inclination shift of light rays caused by the refraction of the windshield in this embodiment can be obtained by using the same processing as in FIG. 1. That is, in the stereo parallax image calibration unit 99, based on the result of performing stereo matching on the left and right images (image P1, image P2) obtained by the matching unit 240 imaging the chart G20 with the left and right cameras (right camera 50, left camera 60), a calibration parameter for calibrating at least the parallax shift caused by the windshield (refraction), which is a light transmission member, is obtained. At this time, the stereo parallax image calibration unit 99 obtains the above-described calibration parameter based on the result of performing stereo matching assuming that different patterns are the same point. At this time, the result of the stereo parallax image calibration unit 99 (calibration data) becomes this parallax shift. The data may be processed to correct this parallax shift, recorded in the stereo parallax calibration data recording unit 98, and calibration may be performed by the stereo parallax image calibration unit 99. By performing such calibration, high accuracy can be achieved for a distant measurement object, but for a nearby measurement object, a parallax shift due to the position shift factor caused by the refraction of the windshield will occur. However, since the allowable value of the parallax shift is larger in the vicinity than in the distance, the influence is small.

[0042] FIG. 7 shows the simulation results of the prior art 1 and the prior art 2. Here, the calculation was performed with the following parameters. <Windshield> ·Radius of curvature -Horizontal: 5.0 m -Vertical: 5.0 m ·Windshield tilt angle: 45 degrees ·Windshield thickness: 5.0 mm ·Windshield refractive index: 1.52 ·Distance between stereo camera and windshield: 50 mm <Stereo camera> ·Baseline length: 200 mm ·Focal length: 4.0 mm ·Sensor pixel pitch: 0.00375 mm ·Viewing angle: -72 degrees to +72 degrees (horizontal), 0 degrees (vertical) <Chart> · Distance between camera and chart: 0.5 m <Object to be measured> · Distance between stereo camera and object to be measured: 50 m · (When estimating the influence of the windshield) Distance between stereo camera and object to be measured: 1000 m

[0043] Figure 7 shows the horizontal angular dependence of the parallax shift. The vertical axis represents the parallax shift, and the horizontal axis represents the horizontal angular field of view. This figure shows the results of two conditions: with and without a windshield. When there is a windshield, the difference between the parallax shift detected using a nearby chart and the parallax shift when the distance between the stereo camera and the object to be measured is 1000 m was calculated. That is, it shows a state where the influence of the inclination shift of the light rays due to the refraction of the windshield has been removed. Therefore, the parallax shift on the vertical axis is the influence of the position shift of the light rays associated with the refraction of the windshield described above. In actual calibration, since the parallax shift due to the inclination shift and position shift factors of the light rays caused by the windshield is detected simultaneously, if the calibration described in Conventional Technique 1 and Conventional Technique 2 is performed, this parallax shift will remain.

[0044] Figure 8 shows the horizontal angular dependence of the ranging distance. The vertical axis represents the ranging distance of the stereo camera, and the horizontal axis represents the horizontal angular field of view. Here, the ranging distance of the stereo camera is calculated using the parallax shift shown in Figure 7. When there is no windshield, accurate ranging (50 m) can be achieved. On the other hand, when there is a windshield, the ranging error increases. Although a ranging error occurs even in a region with a small horizontal angular field of view (for example, a horizontal angular field of view of 0 degrees), the ranging error becomes larger on the wide-angle side.

[0045] Figure 9 shows the horizontal angular dependence of the parallax shift when this embodiment is applied. The vertical axis represents the parallax shift, and the horizontal axis represents the horizontal angular field of view. Similar to Figure 7, it is a state where the influence of the distortion caused by the windshield has been removed. From this, it can be seen that by applying this embodiment, there is almost no parallax shift. Therefore, it can be understood that in this embodiment, the parallax shift due to the distortion factor of the windshield can be obtained with high accuracy.

[0046] Figure 10 shows the horizontal angular dependence of the ranging distance when this embodiment is applied. The vertical axis represents the ranging distance of the stereo camera, and the horizontal axis represents the horizontal angular range. As can be seen, by applying this embodiment, high-precision ranging (50 m) can be achieved.

[0047] Figure 11 shows the horizontal angular dependence of the optimal interval D (in FIG. 6) of the chart (the interval of the periodic pattern or the repeating pattern). The vertical axis represents the interval D, and the horizontal axis represents the horizontal angular range. In FIGS. 9 and 10, the parallax shift occurs at this interval D, and the ranging distance of the stereo camera is calculated. As shown in FIG. 11, the interval D is set to be longer than the baseline length (200 mm) in all horizontal angle regions. Thereby, high-precision results as shown in FIGS. 9 and 10 can be obtained.

[0048] As described above, this embodiment, like the prior art 1 and the prior art 2, compares different regions (performs stereo matching assuming that different patterns are at the same point) and detects the amount of deviation. The difference between this embodiment and the prior art 1 and the prior art 2 is that, as shown in FIG. 6, the horizontal periodic pattern on the chart G20 is a point where the distance D is different from the baseline length B (specifically, larger than the baseline length B). And this embodiment can improve the ranging accuracy by making the horizontal lengths of the two cameras different at least in the first region and the second region.

[0049] Here, in this embodiment, as shown in FIG. 4, there is no difference in the vertical viewing angle. However, for example, as shown in FIG. 12, the amount of change in the width in the horizontal direction (baseline direction) with respect to the vertical viewing angle may be changed. For example, in the regions C13, C23, C43, and C53 (third regions) that are perpendicular to the baseline direction of the sensor surface of the camera in the region C33 (first region) of the chart G20 and the region C33 (first region), the width in the horizontal direction (baseline direction) may be changed. This is because the front glass is tilted, so the optimal distance D in the horizontal direction (baseline direction) with respect to the vertical viewing angle changes. By doing so, the ranging accuracy of the entire image can be improved. And, as shown in FIG. 12, the regions may be connected stepwise (in a stepped manner), or as shown in FIG. 13, the regions may be connected smoothly. And, needless to say, for example, an effect can also be obtained by using two intervals D of a horizontal narrow angle portion and a horizontal wide angle portion.

[0050] Furthermore, in this embodiment, a rectangular chart pattern as shown in FIG. 5 is used, but a circular chart pattern as shown in FIG. 14 may also be used.

[0051] Also, in FIG. 1, the influence of the front glass is corrected using the stereo parallax image correction unit 99 and the stereo parallax correction data recording unit 98, but it is not limited to this. For example, the correction data of the stereo parallax correction data recording unit 98 can also be sent to the affine processing means 20a or the affine processing means 20b for correction.

[0052] [Embodiment 2] Referring to FIG. 15, a calibration method of the image processing apparatus 10 according to Embodiment 2 will be described. The calibration method of Embodiment 1 was the same calculation method as the calibration methods of Conventional Technique 1 and Conventional Technique 2, but it is not limited to this. In this Embodiment 2, a calibration method when considering the influence of the displacement of the light rays due to the refraction of the front glass when the chart is disposed in the vicinity will be described.

[0053] FIG. 15 shows a configuration example of an image processing apparatus 10 including an estimation flow of the influence of the windshield. In this embodiment, as in the first embodiment, it is characterized by using the charts shown in FIGS. 4, 12, and 13. In this embodiment, as shown in FIG. 15, the Y displacement image generation unit 201 is characterized by obtaining a vertical displacement (Y displacement) image of two images. Then, the ΔY displacement image generation unit 80 calculates the difference (ΔY displacement image) between the Y displacement image and the Y displacement image of the reference condition obtained from calculations or the actual machine. Then, the Δ parallax displacement calculation processing unit 85 generates a Δ parallax displacement image by multiplying the ΔY displacement image by a predetermined coefficient. Finally, the parallax displacement calculation processing unit 90 obtains the parallax displacement of the light ray inclination deviation factor due to the refraction of the windshield by adding the Δ parallax displacement image and the parallax displacement image of the reference condition. Here, the Δ parallax displacement and the ΔY displacement indicate the parallax displacement and the Y displacement amount of the reference condition such as the design value or the central value of the product. Here, in order to cope with variations in the radius of curvature of the windshield, the mounting position of the camera with respect to the windshield, the thickness deviation of the windshield, etc., the Δ parallax displacement and the ΔY displacement are used.

[0054] Next, the reason why the parallax displacement can be corrected in this embodiment will be described. Since the windshield has a radius of curvature in the horizontal and vertical directions and a rake angle of the windshield, not only a horizontal change (Δ parallax displacement) but also a vertical change (ΔY displacement) occurs with respect to displacement factors such as variations in the radius of curvature of the windshield, the mounting position of the camera with respect to the windshield, and the thickness deviation of the windshield. Here, there is a correlation between the Δ parallax displacement and the ΔY displacement. Therefore, by detecting the ΔY displacement, the Δ parallax displacement can be obtained. And the parallax displacement can be obtained from the Δ parallax displacement and the parallax displacement of the reference condition. In this embodiment, the parallax displacement data obtained in this way is recorded in the stereo parallax correction data recording unit 98 shown in FIG. 1, and correction is performed by the stereo parallax image correction unit 99.

[0055] As described above, by considering the displacement of the light rays due to the refraction of the windshield when the chart is arranged in the vicinity, it is possible to accurately estimate the influence of the inclination deviation of the light rays due to the refraction of the windshield. Here, in this embodiment, the displacement in the Y direction is used, but it is not limited to this. Needless to say, the same effect can be obtained as long as it is a method of calibrating the influence of the windshield using a chart. When the influence of variation factors such as the mounting position of the windshield and the camera is small, the calibration method of the first embodiment can be used. On the other hand, when the influence of these variation factors is large, it is desirable to use a calibration method as in this embodiment.

[0056] [Summary] As described above, the stereo image processing apparatus 10 of this embodiment includes a stereo matching unit (matching unit 240) that performs stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that image a subject through a light transmission member (windshield 1) to obtain a parallax, and a calibration processing unit (stereo parallax image calibration unit 99) that obtains a calibration parameter for calibrating at least the parallax deviation caused by the light transmission member (windshield 1) based on the result of performing stereo matching on a plurality of images obtained by the stereo matching unit (matching unit 240) imaging a calibration chart with the plurality of cameras. The calibration chart includes a pattern, and the pattern is a repeating pattern larger than the baseline length of the plurality of cameras (FIG. 11). The calibration processing unit (stereo parallax image calibration unit 99) obtains the calibration parameter based on the result of performing stereo matching assuming that different patterns are at the same point.

[0057] In addition, the stereo image processing apparatus 10 of the present embodiment includes a stereo matching unit (matching unit 240) that performs stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that image a subject through a light transmissive member (front glass 1) to obtain a disparity, and a calibration processing unit (stereo disparity image calibration unit 99) that obtains a calibration parameter for calibrating at least a disparity shift caused by the light transmissive member (front glass 1) based on a result of performing stereo matching on a plurality of images obtained by the stereo matching unit (matching unit 240) capturing a calibration chart with the plurality of cameras. The calibration chart includes a pattern, and in the pattern, the length in the baseline direction of the plurality of cameras is different at least between a first region and a second region (in other words, has a first region and a second region where the length in the baseline direction of the plurality of cameras is different) (da2 > da1 in FIG. 4, FIG. 11, etc.). The calibration processing unit (stereo disparity image calibration unit 99) obtains the calibration parameter based on a result of performing stereo matching assuming that different patterns are at the same point in each of the first region and the second region of the calibration chart.

[0058] In addition, when the stereo image processing apparatus 10 of the present embodiment designates a region including a position on a plane including the lens optical axes of the plurality of cameras and on the calibration chart where the perpendicular bisector of the plurality of cameras intersects as the first region (and designates a region where the position in the baseline direction of the plurality of cameras is different from the first region as the second region), the length in the baseline direction of the plurality of cameras in the second region is longer than that in the first region (in other words, the length in the baseline direction of the plurality of cameras becomes longer with respect to the first region as the position in the baseline direction of the plurality of cameras moves away from the first region) (da2 > da1 in FIG. 4, FIG. 11, etc.).

[0059] In addition, there is a third region of the calibration chart in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first region of the calibration chart, and the length in the baseline direction of the plurality of cameras is different between the first region and the third region (FIGS. 12 and 13).

[0060] The stereo image calibration method of this embodiment includes a stereo matching process (matching unit 240) that performs stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that image a subject through a light transmission member (front glass 1) to obtain a parallax, and calibration based on the result of performing stereo matching on a plurality of images obtained by imaging a calibration chart with the plurality of cameras in the stereo matching process (matching unit 240). A calibration process (stereo parallax image calibration unit 99) for obtaining calibration parameters for correcting at least parallax deviation caused by the light transmission member (front glass 1), wherein the calibration chart includes a pattern, and the pattern is a repeating pattern larger than the baseline length of the plurality of cameras (FIG. 11), and in the calibration process (stereo parallax image calibration unit 99), the calibration parameters are obtained based on the result of performing stereo matching assuming that different patterns are at the same point.

[0061] Further, the stereo image calibration method of this embodiment includes a stereo matching process (matching unit 240) that performs stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that image a subject through a light transmission member (front glass 1) to obtain a parallax, and calibration based on the result of performing stereo matching on a plurality of images obtained by imaging a calibration chart with the plurality of cameras in the stereo matching process (matching unit 240). A calibration process (stereo parallax image calibration unit 99) for obtaining calibration parameters for correcting at least parallax deviation caused by the light transmission member (front glass 1), wherein the calibration chart includes a pattern, and the pattern has different lengths in the baseline direction of the plurality of cameras in at least a first region and a second region (in other words, has a first region and a second region with different lengths in the baseline direction of the plurality of cameras) (da2>da1 in FIG. 4, FIG. 11, etc.), and in the calibration process (stereo parallax image calibration unit 99), in each of the first region and the second region of the calibration chart, the calibration parameters are obtained based on the result of performing stereo matching assuming that different patterns are at the same point.

[0062] That is, in the stereo image processing apparatus 10 and the stereo image calibration method of this embodiment, the calibration chart (calibration chart) G20 changes the pattern size from the center to the outside, thereby suppressing the displacement due to the refraction of the light transmission member such as the windshield.

[0063] According to the stereo image processing apparatus 10 and the stereo image calibration method according to this embodiment, it is possible to provide a stereo image processing apparatus 10 and a stereo image calibration method that can calibrate a wide-angle camera with high accuracy without increasing the size of the chart.

[0064] As described above, the embodiments according to the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and even if there are design changes or the like within the scope not departing from the gist of the present invention, they are included in the present invention.

[0065] Note that the present invention is not limited to the above-described embodiments, and various other modified examples are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described.

[0066] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, by an integrated circuit. Further, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program for realizing each function. Information such as a program, table, and file for realizing each function can be stored in a memory, a storage device such as a hard disk or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

[0067] Also, the control lines and information lines show those considered necessary for explanation, and not necessarily all the control lines and information lines are shown on the product. In fact, it may be considered that almost all the components are interconnected.

Explanation of Symbols

[0068] 1…Front glass 10…Stereo camera image processing device (stereo image processing device) 50…Right camera 60…Left camera 20a…Affine processing means 20b…Affine processing means 98…Stereo parallax correction data recording unit 99…Stereo parallax image generation unit (correction processing unit) 200…Stereo parallax image generation unit 201…Y shift image generation unit 400…Road surface cross-section shape estimation unit 500…Stereo parallax three-dimensional object detection unit

Claims

1. A stereo matching unit that performs stereo matching on a plurality of images captured by a plurality of cameras that image a subject through a light transmissive member to obtain a disparity; A calibration processing unit that obtains a calibration parameter for calibrating at least a disparity shift caused by the light transmissive member based on a result of performing stereo matching on a plurality of images obtained by imaging a calibration chart with the plurality of cameras by the stereo matching unit; and The calibration chart includes a pattern; The pattern is a repeating pattern larger than the baseline length of the plurality of cameras; The calibration processing unit obtains the calibration parameter based on a result of performing stereo matching assuming that different patterns are at the same point. A stereo image processing apparatus characterized by the above.

2. A stereo matching unit that performs stereo matching on a plurality of images captured by a plurality of cameras that image a subject through a light transmissive member to obtain a disparity; A calibration processing unit that obtains a calibration parameter for calibrating at least a disparity shift caused by the light transmissive member based on a result of performing stereo matching on a plurality of images obtained by imaging a calibration chart with the plurality of cameras by the stereo matching unit; and The calibration chart includes a pattern; The length of the plurality of cameras in the baseline direction of the pattern is different at least in a first region and a second region; The calibration processing unit obtains the calibration parameter based on a result of performing stereo matching assuming that different patterns are at the same point in each of the first region and the second region of the calibration chart. A stereo image processing apparatus characterized by the above.

3. The stereo image processing apparatus according to claim 2, When a region including a position on a plane including the lens optical axes of the plurality of cameras and on the calibration chart where the perpendicular bisector of the plurality of cameras intersects is defined as the first region, The second region is longer than the first region in the baseline direction of the plurality of cameras. A stereo image processing apparatus characterized by the above.

4. The stereo image processing apparatus according to claim 2, There is a third region of the calibration chart in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first region of the calibration chart, The length of the plurality of cameras in the baseline direction is different between the first region and the third region. A stereo image processing apparatus characterized by the above.

5. A stereo matching process for obtaining a disparity by performing stereo matching on a plurality of images captured by a plurality of cameras that image a subject through a light transmission member; A calibration process for obtaining a calibration parameter for calibrating at least a disparity shift caused by the light transmission member based on a result of performing stereo matching on a plurality of images obtained by imaging a calibration chart with the plurality of cameras in the stereo matching process; and The calibration chart includes a pattern; The pattern is a repeating pattern larger than the baseline length of the plurality of cameras; In the calibration process, the calibration parameter is obtained based on a result of performing stereo matching assuming that different patterns are at the same point. A stereo image calibration method characterized by this.

6. A stereo matching process for obtaining a disparity by performing stereo matching on a plurality of images captured by a plurality of cameras that image a subject through a light transmission member; A calibration process for obtaining a calibration parameter for calibrating at least a disparity shift caused by the light transmission member based on a result of performing stereo matching on a plurality of images obtained by imaging a calibration chart with the plurality of cameras in the stereo matching process; and The calibration chart includes a pattern; The pattern has different lengths in the baseline direction of the plurality of cameras in at least a first region and a second region; In the calibration process, the calibration parameter is obtained based on a result of performing stereo matching assuming that different patterns are at the same point in each of the first region and the second region of the calibration chart. A stereo image calibration method characterized by this.

7. The stereo image calibration method according to claim 6, When a region including a position on a plane including the lens optical axes of the plurality of cameras and on the calibration chart where the perpendicular bisector of the plurality of cameras intersects is defined as the first region, The second region is characterized in that the length in the baseline direction of the plurality of cameras is longer than that of the first region. A stereo image calibration method characterized by this.

8. The stereo image calibration method according to claim 6, There is a third region of the calibration chart in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first region of the calibration chart. A stereo image calibration method, characterized in that the length in the baseline direction of the plurality of cameras is different between the first region and the third region.

Images