Stereo image processing device and stereo image calibration method

Abstract

The objective of the present invention is to provide a stereo image processing device and a stereo image calibration method capable of calibrating a wide-angle camera mounted in a vehicle with a high degree of accuracy without increasing the size of a chart. A calibration chart includes patterns, wherein: the lengths of the patterns in the baseline direction of a plurality of cameras differ at least between a first region and a second region; and a calibration processing unit obtains a calibration parameter on the basis of a result of performing stereo matching in each of the first region and the second region of the calibration chart, assuming that the different patterns are the same point.

Description

Stereo image processing device and stereo image calibration method

[0001] The present invention relates to an (in-vehicle) stereo image processing device mounted on, for example, a vehicle, and a stereo image calibration method.

[0002] A stereo camera is known as a device for recognizing objects three-dimensionally. A stereo camera uses the differences in the images captured by multiple cameras placed at different positions to detect parallax between the multiple cameras based on trigonometry, and then uses this parallax to detect the depth and position of an object, thereby enabling accurate detection of the position of the measurement target.

[0003] Such stereo cameras are mounted on automobiles and other vehicles and are used in in-vehicle sensing technology to detect the positions of obstacles, etc. In order to accommodate many use cases, in-vehicle sensing technology is required to detect obstacles, etc. with a wide angle of view and to detect obstacles at greater distances (wider angle of view and longer range).

[0004] In-vehicle stereo cameras are generally installed inside the vehicle to avoid the effects of dirt, etc., but as the angle of view becomes wider, the effects of the vehicle's windshield become unignorable. Conventionally, a calibration process called aiming is performed on the stereo camera during vehicle manufacturing and inspection to correct for any misalignment in the stereo camera, but further measures are required to widen the angle of view.

[0005] Patent Document 1 is known as a method for calibrating a stereo camera that takes into account the influence of a windshield. Patent Document 1 describes the problem as "to provide a stereo camera calibration device and calibration method that can appropriately calibrate a stereo camera using a target board even in a narrow space," and as a solution, describes the solution as "a stereo camera calibration device that measures the distance to an object based on parallax, the device comprising: a stereo camera 3 having a pair of cameras 4 and 6 spaced apart by a predetermined base line length B; a target board 8 that is arranged parallel to the arrangement direction of the pair of cameras 4 and 6 and includes at least two targets 8a and 8b spaced apart by the same distance as the base line length B; and a computing device that assumes that the target board 8 is located at infinity relative to the stereo camera 3, treats the two targets 8a and 8b included in images of the target board 8 captured by each of the pair of cameras 4 and 6 as the same target, and calibrates the parallax shift of the pair of cameras 4 and 6." Patent Document 2 also describes a similar calibration technique. Japanese Patent Laid-Open No. 2007-111222 describes the problem as "variably setting a search range for stereo matching according to a position on an image," and as a solution, states that "the system includes a comparison image line memory 7, an address generation circuit 10, and a stereo matching circuit 8. The line memory 7 stores image data within a reference pixel region in one captured image and image data on a horizontal line corresponding to the vertical position of the reference pixel region in the other captured image. The address generation circuit 10 sets the search range for stereo matching and instructs the line memory 7 to read image data within the set search range and image data within the reference image region. The stereo matching circuit 8 identifies the correlated point of the reference pixel region by stereo matching, based on the image data within the search range read from the line memory 7 and the image data of the reference pixel region. Here, the address generation circuit 10 calibrates the position of the search range for the reference pixel region, based on the degree of deviation of corresponding points at infinity relative to the horizontal position of the reference pixel region."

[0006] JP 2017-62150 A JP 2001-92968 A

[0007] Patent Documents 1 and 2 state that the effects of shifts in the inclination of light rays due to refraction by the windshield can be calibrated by placing a chart near a stereo camera, matching the interval (baseline length) between the two cameras of the stereo camera with the period of the chart pattern, and virtually treating the chart as being at infinity. However, these technologies do not take into account the effects of shifts in the position of light rays due to refraction by the windshield, and therefore have the problem that complete calibration cannot be achieved when a chart is actually placed near the stereo camera.

[0008] The present invention has been made in consideration of the above-mentioned problems, and provides a stereo image processing device and a stereo image calibration method that can calibrate a wide-angle stereo camera with high precision by placing a chart near the stereo camera without increasing the size of the chart.

[0009] One aspect of the stereo image processing device according to the present invention includes a stereo matching unit that performs stereo matching of multiple images captured by multiple cameras that capture images of a subject through a light-transmitting member to determine parallax, and a calibration processing unit that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member based on the results of stereo matching of multiple images obtained by the stereo matching unit capturing images of a calibration chart with the multiple cameras, wherein the calibration chart includes a pattern, and the pattern is a repeated pattern that is larger than a baseline length of the multiple cameras, and the calibration processing unit determines the calibration parameters based on the results of stereo matching assuming that different patterns are the same point.

[0010] According to the stereo image processing device and stereo image calibration method of the present invention, it is possible to provide a stereo image processing device and stereo image calibration method that can calibrate a wide-angle camera with high precision without increasing the size of the chart.

[0011] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0012] 1 is a block diagram illustrating a configuration of a stereo camera image processing device according to Example 1. FIG. 2 is a diagram illustrating a problem of the conventional method according to Example 1. FIG. 3 is another diagram illustrating a problem of the conventional method according to Example 1. FIG. 4 shows the shape of a chart area according to Example 1. FIG. 5 shows a pattern in each area of ​​the chart according to Example 1. FIG. 6 shows an optical path according to Example 1. FIG. 7 shows parallax shift according to the conventional method according to Example 1. FIG. 8 shows the distance measurement according to the conventional method according to Example 1. FIG. 9 shows parallax shift according to Example 1. FIG. 10 shows the distance measurement according to Example 1. FIG. 11 shows the interval D according to Example 1. FIG. 12 shows another shape of the chart area according to Example 1. FIG. 13 shows another shape of the chart area according to Example 1. FIG. 14 shows another pattern of each area of ​​the chart according to Example 1. FIG. 15 is a block diagram illustrating a configuration for creating calibration data of a stereo camera image processing device according to Example 2.

[0013] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings illustrate embodiments in accordance with the principles of the present disclosure, but these drawings are for the purpose of understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0014] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to practice the present disclosure, it should be understood that other embodiments are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0015] [Example 1] With reference to Fig. 1, the configuration of a stereo camera image processing device 10 (hereinafter referred to as "image processing device 10" or "stereo image processing device 10") according to Example 1 will be described. This image processing device 10 is mounted on a vehicle such as an automobile, and is used to detect the distance from the vehicle to three-dimensional objects (other automobiles, buildings, pedestrians, etc.) around the vehicle. The following description will be given using an example in which the image processing device 10 is mounted on a vehicle, but the present invention is not limited to this.

[0016] 1 is a block diagram showing an example of the configuration of an image processing device 10 according to a first embodiment. The image processing device 10 is configured to detect surrounding three-dimensional objects based on images obtained by a right camera 50 and a left camera 60, and to issue an alarm as necessary. The right camera 50 and the left camera 60 form a stereo camera. Note that the number of cameras that form the stereo camera is not limited to two, a left camera and a right camera.

[0017] As an example, this image processing device 10 is configured to include an image processing unit 100, a stereo parallax image generation unit 200, a stereo parallax image calibration unit 99, a road surface cross-sectional shape estimation unit 400, a stereoscopic three-dimensional object detection unit 500, and an alarm control unit 700.

[0018] In this image processing device 10, in an area that can be captured in common by the right camera 50 and the left camera 60 (hereinafter referred to as the "stereoscopic viewing area"), a stereoscopic parallax image is generated in the stereoscopic parallax image generation unit 200 by utilizing the parallax between the right camera 50 and the left camera 60. Then, the stereoscopic three-dimensional object detection unit 500 measures the distance from the vehicle to the three-dimensional object according to the parallax.

[0019] Although not shown, the right camera 50 and the left camera 60 each include a lens and an image sensor. The right camera 50 and the left camera 60 each capture (take) an image of an object via a lens using the image sensor. In this embodiment, the right camera 50 and the left camera 60 are mounted inside the vehicle and capture images of objects around the vehicle through a light-transmitting member such as the windshield. The image processing device 10 captures an image P1 (first image) from the right camera 50 and an image P2 (second image) from the left camera 60.

[0020] The image processing unit 100 is configured to include, for example, affine processing means 20 a, 20 b, luminance correction means 21 a, 21 b, pixel interpolation means 22 a, 22 b, and luminance information generation means 23 a, 23 b. The image processing unit 100 applies predetermined image processing to the images P1 and P2 obtained by the right camera 50 and the left camera 60, and supplies the images to the stereo parallax image generation unit 200.

[0021] The affine processing means 20a applies affine processing to the image P1 from the right camera 50. The affine processing is, for example, a linear coordinate transformation process, but may also include non-linear calculations. As a result of this affine processing, the affine processing means 20a acquires an image P3 (third image). Similarly, the affine processing means 20b applies affine processing to the image P2 from the left camera 60 to acquire an image P4 (fourth image).

[0022] The affine processing means 20a and 20b may also perform distortion transformation processing other than affine processing. In this embodiment, f sin θ, the projection method of the fisheye lens, is projectively transformed into a coordinate system of (f tan θx, f tan θy). Here, f is the focal length of the fisheye lens, θ is the angle of view incident on the fisheye lens, and θx and θy are the horizontal and vertical components of the angle of view incident on the fisheye lens. Furthermore, in this embodiment, the affine processing means 20a and 20b calibrate vertical pixel displacement due to the influence of the windshield.

[0023] The luminance correction means 21a corrects the luminance of each pixel in the image P3. For example, the luminance of each pixel in the image P3 is corrected based on the gain of the right camera 50, the difference in gain of each pixel in the image P3, etc. Similarly, the luminance correction means 21b corrects the luminance of each pixel in the image P4.

[0024] The pixel interpolation unit 22a performs demosaicing on the image P3, for example, converting the image from a raw image to a color image. Similarly, the pixel interpolation unit 22b performs demosaicing on the image P4.

[0025] The luminance information generating means 23a generates luminance information for the image P3. For example, the luminance information generating means 23a converts information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generating means 23b generates luminance information for the image P4.

[0026] The stereo parallax image generating unit 200 generates a stereo parallax image of the stereo viewing region by using the image of the stereo viewing region (common viewing region) from among the obtained images P3 and P4.

[0027] The stereo parallax image generation unit 200 includes an exposure adjustment unit 210 and a sensitivity correction unit 220, and can execute feedback control of the exposure amount, sensitivity, etc. of the right camera 50 and the left camera 60 to the right camera 50 and the left camera 60. The stereo parallax image generation unit 200 further includes a geometric correction unit 230 that performs geometric correction of the left and right images, and a matching unit 240 that performs matching processing of the left and right images. The matching unit 240 performs stereo matching of the left and right images to obtain parallax (stereo parallax images).

[0028] The stereo parallax image calibration unit 99 calibrates the parallax deviation caused by a light-transmitting member such as a windshield, based on the data previously acquired from the stereo parallax calibration data recording unit 98 .

[0029] The road surface cross-sectional shape estimation unit 400 estimates the cross-sectional shape of the road surface of the road along which the vehicle on which the image processing device 10 is mounted is scheduled to travel.

[0030] The stereoscopic three-dimensional object detection unit 500 detects three-dimensional objects in the stereoscopic field according to the stereo disparity images generated by the stereo disparity image generation unit 200. Furthermore, stereo matching is applied to the detected three-dimensional objects to detect disparity and identify the type of three-dimensional object (e.g., pedestrian, bicycle, vehicle, building). By detecting and identifying the type of three-dimensional object, the type to be used for preventive safety can be further identified. When a vehicle is detected, the detection result can be used for preceding vehicle tracking control and emergency braking control. If the detected three-dimensional object is a pedestrian or bicycle, emergency braking control or warning control can be performed. Compared to stationary objects, objects that jump out toward the vehicle are subject to warnings and control for objects within a wider field of view. By measuring the distance to these detected objects and estimating the moving speed of the objects being tracked over time, the warning control unit 700 can provide more appropriate warnings and control.

[0031] In FIG. 1, this embodiment is characterized by creating data for a stereo parallax calibration data recording unit 98 .

[0032] Next, the effects of this embodiment will be described. First, the problems of the conventional technology will be described in detail. As described above, in Patent Document 1 (hereinafter referred to as "Prior Art 1") and Patent Document 2 (hereinafter referred to as "Prior Art 2"), a chart is placed near a stereo camera, the distance between the two cameras of the stereo camera (baseline length) and the period of the chart pattern are made to match the base line length, and the chart is virtually at infinity, thereby calibrating the effect of the tilt shift of light rays due to refraction by the windshield. However, there are two main effects of refraction by the windshield. The first is image distortion due to the tilt shift of light rays. This image distortion occurs equally for charts placed nearby and charts placed far away. The second is positional deviation of light rays. A characteristic of the light positional deviation is that the effect is greater for charts and measurement objects placed nearby, but is negligible for charts and measurement objects placed far away.

[0033] Of these two factors, Prior Art 1 and Prior Art 2 do not take into account the effect of light beam positional deviation due to refraction by the windshield. As described above, this light beam positional deviation is detected when measuring the distance to a nearby measurement object, but can be ignored when detecting a distant measurement object. Typically, parallax deviation is calibrated to optimize distance measurement. On the other hand, if parallax deviation is calibrated using a nearby chart, parallax deviation occurs when detecting a distant measurement object due to light beam positional deviation due to refraction by the windshield. For this reason, Prior Art 1 and Prior Art 2 have the problem that parallax deviation cannot be completely calibrated when a chart is placed nearby.

[0034] FIG. 2 is a diagram illustrating this problem. This diagram shows a cross section connecting the lens pupil positions of the right camera 50 and the left camera 60. For ease of explanation, the amount of light ray displacement due to the windshield 1 is exaggerated. In this diagram, the light rays at a horizontal angle of view of 0 degrees for the right camera 50 and the left camera 60 are designated as light rays R51 and R61, respectively. The distance between the light rays R61 and R51 incident on the windshield 1 (the light rays R61 and R51 on the opposite side of the windshield 1 from the stereo camera image processing device 10) is designated as distance D1. The light rays on the horizontal wide-angle side for the right camera 50 and the left camera 60 are designated as light rays R52 and R62. Similarly, the distance between the light rays R62 and R52 incident on the windshield 1 is designated as distance D2. Furthermore, the line extending the light ray R51 between the windshield 1 and the right camera 50 is defined as axis K51, and the line extending the light ray R61 between the windshield 1 and the left camera 60 is defined as axis K61. Similarly, the line extending the light ray R52 between the windshield 1 and the right camera 50 is defined as axis K52, and the line extending the light ray R62 between the windshield 1 and the left camera 60 is defined as axis K62. Here, the distance between the axis K51 and the axis K61 and the distance between the axis K52 and the axis K62 coincide with the base length B.

[0035] First, consider a horizontal angle of view of 0 degrees. Ray R51 and axis K51 are almost the same, but do not completely coincide. The same is true for ray R61 and axis K61. As a result, base length B and distance D1 do not coincide. Meanwhile, distance D2 at a horizontal wide angle also differs from base length B. What is distinctive here is that distance D1 and distance D2 differ due to the influence of windshield 1. Prior art 1 and prior art 2 estimate the influence of distortion caused by the windshield by utilizing the fact that distance D1, distance D2, and base length B coincide. However, in reality, distance D1, distance D2, and base length B do not coincide.

[0036] FIG. 3 is a diagram illustrating the influence of differences in the baseline length B and the distance D (distance D1, distance D2). Like FIG. 2, this diagram also shows a cross section connecting the lens pupil positions of the right camera 50 and the left camera 60. For ease of explanation, the amount of light beam displacement due to the windshield 1 is exaggerated. The charts described in Prior Art 1 and Prior Art 2 use a pattern with the same period as the baseline length. This diagram also uses a chart G10 with a similar pattern period that is the same as the baseline length. In this diagram, axes K53 and K63 are the axes of a predetermined angle of view from the pupil positions of the right camera 50 and the left camera 60. The points of contact between the chart G10, the axis K53, and the axis K63 are positions Q1 and Q2, respectively. Because the inclinations of the axis K53 and the axis K63 are equal, the distance between the positions Q1 and Q2 is the baseline length B. This configuration is the same as that of Prior Art 1 and Prior Art 2. The light rays detected by the right camera 50 and the left camera 60 at positions Q1 and Q2 are defined as light rays R53 and R63. The lines extending light rays R53 and R63 between the pupil positions of the right camera 50 and the left camera 60 and the windshield 1 are defined as lines S53 and S63, respectively, and the intersection of lines S53 and S63 is defined as position TP1.

[0037] Prior arts 1 and 2 claim that virtual infinity can be detected by using a pattern on chart G10 with the same period as the baseline length. To achieve this, two axes, such as axis K53 and axis K63, must be parallel. However, the angle of light ray R53 that passes through windshield 1 and enters right camera 50 is different from the angle of light ray R63 that enters left camera 60. For this reason, a stereo camera calibrated with this chart will generate a large error. In this diagram, what should be infinity will be detected as if the chart were at position TP1, resulting in a major problem: the camera will be calibrated at distance L.

[0038] This embodiment takes this into consideration. Figure 4 shows the shapes of the regions of chart G20 in this embodiment. Note that each region has a similar pattern (a rectangular shape in the illustrated example). In this embodiment, each region (pattern) of chart G20 is characterized by a different width in the horizontal direction (baseline direction). For example, the horizontal width da1 of region C33 (first region) of chart G20 is different from the horizontal width da2 of region C35 (second region). Width da2 is larger than width da1. In this figure, when the plane including the lens optical axes of the two cameras of the stereo camera and the perpendicular bisectors of the two cameras are assumed to be in area C33 on the chart G20 (in other words, when the plane including the lens optical axes of the two cameras and the position on the chart G20 where the perpendicular bisectors of the two cameras intersect is defined as area C33, or when the plane including the chart G20 and the lens optical axes of the two cameras and the position where the perpendicular bisectors of the two cameras intersect is defined as area 33, and when the horizontal (baseline) position of the plane including the chart G20 and the lens optical axes of the two cameras is defined as area 35), both widths da1 and da2 are larger than the baseline length B, and width da1 is closer to baseline length B than width da2 (i.e., width da2 is larger than width da1). Note that as the radius of curvature of the windshield becomes smaller, widths da1 and da2 become farther apart from baseline length B (see FIG. 11 ). On the other hand, as the radius of curvature of the windshield increases, the widths da1 and da2 approach the baseline length B (see FIG. 11). Because the radius of curvature of the windshield varies depending on the vehicle, the chart spacing must be adjusted accordingly. Furthermore, if the distance between the stereo camera and the chart is small, the widths da1 and da2 become farther away from the baseline length B. On the other hand, as the distance between the stereo camera and the chart increases, the widths da1 and da2 approach the baseline length B.

[0039] FIG. 5 shows patterns within each region of the chart G20 in FIG. 4 . For example, the pattern within region C33 on the chart G20 is shown. Here, too, the horizontal widths of the regions are different. For example, the horizontal width db1 of (0,0) is different from the horizontal width db2 of (2,0). Width db2 is larger than width db1. The detected light intensity of each region is set to be different, preventing incorrect matching. With this configuration, for example, the amount of deviation between corresponding points in an image in which region C34 of the chart G20 is detected by the right camera 50 and an image in which region C33 is detected by the left camera 60 can be detected. The amount of deviation can be calculated using detection methods such as block matching and feature point extraction used in stereo cameras.

[0040] Figure 6 shows the light paths when the positional deviation of light rays due to the windshield is taken into consideration. First, the light rays from the left camera 60 are the same as those in Figure 3. On the other hand, the angle of light ray R54 is changed so that the angle of light ray R63 incident on the left camera 60 matches the angle of light ray R54 incident on the right camera 50 (lines S54 and S63 are parallel, and the distance between them is the base length B). In this case, the horizontal periodic pattern on the chart G20 (in other words, the distance between positions Q3 and Q2 of the images on the chart G20 detected by the right camera 50 and left camera 60 using light ray R54 and light ray R63) is spaced apart by a distance D.

[0041] In this embodiment, the parallax shift caused by the inclination shift of light rays due to refraction by the windshield can be calculated using a process equivalent to that shown in FIG. 1 . That is, in the stereo parallax image calibration unit 99, the matching unit 240 performs stereo matching on left and right images (images P1 and P2) obtained by capturing images of the chart G20 using left and right cameras (right camera 50 and left camera 60). Based on the results, the stereo parallax image calibration unit 99 calculates the calibration parameters described above based on the results of stereo matching assuming that different patterns are the same point. The results of the calibration data of the stereo parallax image calibration unit 99 represent this parallax shift. Data is processed to correct this parallax shift, recorded in the stereo parallax calibration data recording unit 98, and calibration is then performed by the stereo parallax image calibration unit 99. While such calibration improves accuracy for distant measurement objects, parallax shift due to refraction by the windshield occurs for nearby measurement objects. However, since the allowable value of parallax displacement is larger in the near field than in the far field, the effect is small.

[0042] Figure 7 shows the simulation results for Conventional Technique 1 and Conventional Technique 2. Here, calculations were performed using the following parameters. <Windshield> Radius of curvature - Horizontal: 5.0 m - Vertical: 5.0 m Windshield tilt angle: 45 degrees Windshield thickness: 5.0 mm Windshield refractive index: 1.52 Distance between stereo camera and windshield: 50 mm <Stereo camera> Baseline length: 200 mm Focal length: 4.0 mm Sensor pixel pitch: 0.00375 mm Angle of view: -72 degrees to +72 degrees (horizontal), 0 degrees (vertical) <Chart> Distance between camera and chart: 0.5 m <Measurement object> Distance between stereo camera and measurement object: 50 m Distance between stereo camera and measurement object (when estimating the effect of the windshield): 1000 m

[0043] FIG. 7 shows the dependency of parallax shift on the horizontal angle of view. The vertical axis represents the parallax shift, and the horizontal axis represents the horizontal angle of view. This figure shows the results for two conditions: with and without a windshield. Note that, for the case with a windshield, the difference between the parallax shift detected using the proximity chart and the parallax shift when the distance between the stereo camera and the measurement object was set to 1,000 m was calculated. In other words, this shows the state in which the effect of the inclination shift of light rays due to refraction by the windshield has been removed. Therefore, the parallax shift on the vertical axis is the effect of the position shift of light rays due to the refraction by the windshield. In actual calibration, the inclination shift of light rays due to the windshield and the parallax shift, which is a cause of the position shift, are detected simultaneously. Therefore, when the calibrations described in Prior Art 1 and Prior Art 2 are performed, this parallax shift remains.

[0044] Figure 8 shows the dependency of the measured distance on the horizontal angle of view. The vertical axis represents the measured distance of the stereo camera, and the horizontal axis represents the horizontal angle of view. Here, the measured distance of the stereo camera is calculated using the parallax displacement shown in Figure 7. When there is no windshield, accurate distance measurement (50 m) is achieved. On the other hand, when there is a windshield, the distance measurement error increases. Although distance measurement errors occur even in areas with small horizontal angles of view (for example, a horizontal angle of view of 0 degrees), the distance measurement error increases on the wide-angle side.

[0045] FIG. 9 shows the horizontal angle of view dependency of parallax shift when this embodiment is applied. The vertical axis represents parallax shift, and the horizontal axis represents horizontal angle of view. As in FIG. 7, this is a state in which the influence of distortion due to the windshield has been removed. This shows that by applying this embodiment, there is almost no parallax shift. Therefore, it can be seen that this embodiment can accurately determine the parallax shift caused by distortion due to the windshield.

[0046] 10 shows the dependency of the measured distance on the horizontal angle of view when this embodiment is applied. The vertical axis represents the measured distance of the stereo camera, and the horizontal axis represents the horizontal angle of view. As can be seen, by applying this embodiment, highly accurate distance measurement (50 m) can be achieved.

[0047] FIG. 11 shows the horizontal angle of view dependency of the optimal chart spacing D (FIG. 6) (spacing of periodic or repetitive patterns). The vertical axis represents spacing D, and the horizontal axis represents horizontal angle of view. In FIGS. 9 and 10, the parallax shift and the distance measured by the stereo camera are calculated using this spacing D. As shown in FIG. 11, the spacing D is set to be longer than the base line length (200 mm) in all horizontal angle ranges. This allows for highly accurate results as shown in FIGS. 9 and 10.

[0048] As described above, this embodiment, like Prior Art 1 and Prior Art 2, compares different areas (performs stereo matching by assuming that different patterns are the same point) and detects the amount of deviation. The difference between this embodiment and Prior Art 1 and Prior Art 2 is that, as shown in FIG. 6 , the distance D of the horizontal periodic pattern on chart G20 is different from the base line length B (more specifically, is greater than the base line length B). This embodiment can improve distance measurement accuracy by making the horizontal lengths of the two cameras different in at least the first and second areas.

[0049] While the present embodiment does not differ in the vertical angle of view as shown in FIG. 4 , the amount of change in the horizontal (baseline) width may be varied with respect to the vertical angle of view, as shown in FIG. 12 . For example, the horizontal (baseline) width may be varied between area C33 (first area) on chart G20 and areas C13, C23, C43, and C53 (third areas) perpendicular to the baseline of the camera sensor surface of area C33 (first area). This is because the windshield is tilted, so the optimal horizontal (baseline) spacing D varies with the vertical angle of view. This improves the ranging accuracy of the entire image. The areas may be connected in stages (step-like) as shown in FIG. 12 , or smoothly connected as shown in FIG. 13 . It goes without saying that the same effect can be achieved by using two spacings D, for example, one for the horizontal narrow-angle section and one for the horizontal wide-angle section.

[0050] Furthermore, in this embodiment, a rectangular chart pattern as shown in FIG. 5 is used, but a circular chart pattern as shown in FIG. 14 may also be used.

[0051] 1, the influence of the windshield is calibrated using the stereo parallax image calibration unit 99 and the stereo parallax calibration data recording unit 98, but this is not limiting. For example, calibration can also be performed by sending the calibration data of the stereo parallax calibration data recording unit 98 to the affine processing means 20a or 20b.

[0052] [Example 2] A calibration method for the image processing device 10 according to Example 2 will be described with reference to Fig. 15. The calibration method in Example 1 uses the same calculation method as the calibration methods in Prior Art 1 and Prior Art 2, but is not limited to this. In Example 2, a calibration method will be described that takes into account the influence of light misalignment caused by refraction at the windshield when a chart is placed nearby.

[0053] FIG. 15 shows an example configuration of the image processing device 10, including a flow for estimating the influence of a windshield. Similar to the first embodiment, this embodiment is characterized by using the charts shown in FIGS. 4, 12, and 13. As shown in FIG. 15, this embodiment is characterized by a Y-displacement image generation unit 201 calculating a vertical displacement (Y-displacement) image between two images. Then, a ΔY-displacement image generation unit 80 calculates the difference (ΔY-displacement image) between the Y-displacement image and a Y-displacement image under reference conditions obtained by calculation or from an actual device. Then, a Δ-parallax displacement calculation processing unit 85 generates a Δ-parallax displacement image by multiplying the ΔY-displacement image by a predetermined coefficient. Finally, a parallax displacement calculation processing unit 90 adds the Δ-parallax displacement image and the parallax displacement image under reference conditions to calculate the parallax displacement, which is a cause of the inclination deviation of light rays due to refraction by the windshield. Here, the Δ-parallax displacement and the ΔY-displacement represent the parallax displacement and Y-displacement amounts under reference conditions, such as design values ​​or product center values. Here, Δ parallax shift and ΔY shift are used to accommodate variations in the radius of curvature of the windshield, the mounting position of the camera relative to the windshield, and variations in the thickness of the windshield.

[0054] Next, the reason why the parallax displacement can be calibrated in this embodiment will be explained. Because a windshield has a radius of curvature in both the horizontal and vertical directions and a tilt angle of the windshield, displacement factors such as variations in the radius of curvature of the windshield, the mounting position of the camera relative to the windshield, and variations in the thickness of the windshield cause not only a change in the horizontal direction (Δ parallax displacement) but also a change in the vertical direction (ΔY displacement). Here, there is a correlation between the Δ parallax displacement and the ΔY displacement. Therefore, the Δ parallax displacement can be calculated by detecting the ΔY displacement. Then, the parallax displacement can be calculated from the Δ parallax displacement and the parallax displacement under the reference conditions. In this embodiment, the parallax displacement data calculated in this manner is recorded in the stereo parallax calibration data recording unit 98 shown in FIG. 1, and calibration is performed by the stereo parallax image calibration unit 99.

[0055] As described above, by taking into account the positional deviation of light rays due to refraction by the windshield when a chart is placed nearby, it is possible to estimate with high accuracy the influence of the tilt deviation of light rays due to refraction by the windshield. While the deviation in the Y direction is used in this embodiment, this is not limited to this, and it goes without saying that a similar effect can be obtained as long as a chart is used to calibrate the influence of the windshield. Note that the calibration method of Example 1 can be used when the influence of variation factors such as the windshield and camera mounting position is small. On the other hand, when the influence of these variation factors is large, it is desirable to use a calibration method such as that of this embodiment.

[0056] [Summary] As described above, the stereo image processing device 10 of this embodiment includes a stereo matching unit (matching unit 240) that determines parallax by performing stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that capture images of a subject through a light-transmitting member (windshield 1), and a calibration processing unit (stereo parallax image calibration unit 99) that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on the results of stereo matching performed by the stereo matching unit (matching unit 240) on a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras, where the calibration chart includes a pattern, and the pattern is a repeated pattern that is larger than the baseline lengths of the plurality of cameras ( FIG. 11 ), and the calibration processing unit (stereo parallax image calibration unit 99) determines the calibration parameters based on the results of stereo matching assuming that different patterns are the same point.

[0057] The stereo image processing device 10 of this embodiment also includes a stereo matching unit (matching unit 240) that performs stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that capture images of a subject through a light-transmitting member (windshield 1) to determine parallax, and a calibration processing unit (stage 1) that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on the results of stereo matching performed by the stereo matching unit (matching unit 240) on a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras. and a stereo parallax image calibration unit 99), wherein the calibration chart includes a pattern, and the length of the pattern in the baseline direction of the plurality of cameras is different at least between a first region and a second region (in other words, the pattern has a first region and a second region in which the lengths of the baseline directions of the plurality of cameras are different) (da2>da1 in FIG. 4, FIG. 11, etc.), and the calibration processing unit (stereo parallax image calibration unit 99) obtains the calibration parameters based on a result of stereo matching in the first region and the second region of the calibration chart, assuming that different patterns are the same point.

[0058] Furthermore, in the stereo image processing device 10 of this embodiment, when the first region is defined as an area on a plane including the lens optical axes of the multiple cameras and including a position on the calibration chart where the perpendicular bisectors of the multiple cameras intersect (and the second region is defined as an area where the baseline positions of the multiple cameras differ from the first region), the length of the second region in the baseline direction of the multiple cameras becomes longer than the first region (in other words, the further the baseline positions of the multiple cameras are from the first region, the longer the length of the baseline direction of the multiple cameras becomes relative to the first region) (da2>da1 in Figure 4, Figure 11, etc.).

[0059] Furthermore, in the stereo image processing device 10 of this embodiment, a third area of ​​the calibration chart is located in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first area of ​​the calibration chart, and the lengths of the baseline directions of the multiple cameras are different between the first area and the third area (FIGS. 12 and 13).

[0060] The stereo image calibration method of this embodiment includes: a stereo matching process (matching unit 240) that determines parallax by performing stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that capture images of an object through a light-transmitting member (windshield 1); and a calibration process (stereo parallax image calibration unit 99) that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on the results of stereo matching on a plurality of images obtained by capturing images of a calibration chart with the plurality of cameras in the stereo matching process (matching unit 240). The calibration chart includes a pattern, and the pattern is a repeated pattern that is larger than the baseline lengths of the plurality of cameras ( FIG. 11 ). In the calibration process (stereo parallax image calibration unit 99), the calibration parameters are determined based on the results of stereo matching performed on the basis that different patterns are the same point.

[0061] The stereo image calibration method of this embodiment includes a stereo matching process (matching unit 240) for determining a parallax by performing stereo matching on a plurality of images captured by a plurality of cameras (right camera 50, left camera 60) that capture images of a subject through a light-transmitting member (windshield 1), and a calibration process (stereo matching unit 240) for determining a calibration parameter for calibrating at least the parallax shift caused by the light-transmitting member (windshield 1) based on the results of the stereo matching on a plurality of images obtained by capturing images of a calibration chart by the plurality of cameras in the stereo matching process (matching unit 240). a parallax image calibration unit 99), wherein the calibration chart includes a pattern, and the length of the pattern in the baseline direction of the plurality of cameras is different at least between a first region and a second region (in other words, the pattern has a first region and a second region in which the lengths of the baseline directions of the plurality of cameras are different) (da2>da1 in FIG. 4, FIG. 11, etc.), and in the calibration process (stereo parallax image calibration unit 99), the calibration parameters are obtained based on a result of stereo matching in the first region and the second region of the calibration chart, where different patterns are considered to be the same point.

[0062] In other words, the stereo image processing device 10 and stereo image calibration method of this embodiment suppress positional deviation due to refraction of light-transmitting members such as windshields by changing the pattern size of the calibration chart (calibration chart) G20 from the center to the outside.

[0063] According to the stereo image processing device 10 and stereo image calibration method of this embodiment, it is possible to provide a stereo image processing device 10 and stereo image calibration method that can calibrate a wide-angle camera with high precision without increasing the size of the chart.

[0064] Although the embodiments of the present invention have been described in detail above using the drawings, the specific configuration is not limited to these embodiments, and even if there are design changes and the like within the scope of the present invention, they are also included in the present invention.

[0065] It should be noted that the present invention is not limited to the above-described embodiment, and includes various other modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations.

[0066] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0067] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0068] 1... Windshield 10... Stereo camera image processing device (stereo image processing device) 50... Right camera 60... Left camera 20a... Affine processing means 20b... Affine processing means 98... Stereo parallax calibration data recording unit 99... Stereo parallax image generating unit (calibration processing unit) 200... Stereo parallax image generating unit 201... Y-deviation image generating unit 400... Road surface cross-sectional shape estimating unit 500... Stereoscopic three-dimensional object detecting unit

Claims

1. A stereo image processing device comprising: a stereo matching unit that performs stereo matching of multiple images captured by multiple cameras that capture an image of a subject through a light-transmitting member to determine parallax; and a calibration processing unit that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member based on the results of stereo matching of multiple images obtained by the stereo matching unit capturing images of a calibration chart with the multiple cameras, wherein the calibration chart includes a pattern, and the pattern is a repeating pattern that is larger than the baseline length of the multiple cameras, and the calibration processing unit determines the calibration parameters based on the results of stereo matching assuming that different patterns are the same point.

2. A stereo image processing device comprising: a stereo matching unit that performs stereo matching of multiple images captured by multiple cameras that capture an image of a subject through a light-transmitting member to determine parallax; and a calibration processing unit that determines calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member based on the results of stereo matching of multiple images obtained by the stereo matching unit capturing images of a calibration chart with the multiple cameras, wherein the calibration chart includes a pattern, and the length of the pattern in the baseline direction of the multiple cameras is different at least in a first area and a second area, and the calibration processing unit determines the calibration parameters based on the results of stereo matching in the first area and the second area of ​​the calibration chart, assuming that different patterns are the same point.

3. A stereo image processing device as described in claim 2, characterized in that when the first area is defined as an area on a plane including the lens optical axes of the multiple cameras and including a position on the calibration chart where the perpendicular bisectors of the multiple cameras intersect, the second area has a longer length in the baseline direction of the multiple cameras than the first area.

4. A stereo image processing device as described in claim 2, characterized in that a third area of ​​the calibration chart is located in a direction perpendicular to the baseline direction of the sensor surface of the camera of the first area of ​​the calibration chart, and the lengths of the baseline direction of the multiple cameras are different between the first area and the third area.

5. A stereo image calibration method comprising: a stereo matching process for determining parallax by performing stereo matching of multiple images captured by multiple cameras that capture an image of a subject through a light-transmitting member; and a calibration process for determining calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member based on the results of the stereo matching of multiple images obtained by capturing images of a calibration chart with the multiple cameras in the stereo matching process, wherein the calibration chart includes a pattern, and the pattern is a repeated pattern that is larger than the baseline length of the multiple cameras, and the calibration process determines the calibration parameters based on the results of stereo matching performed assuming that different patterns are the same point.

6. A stereo image calibration method comprising: a stereo matching process for determining parallax by performing stereo matching of multiple images captured by multiple cameras that capture an image of a subject through a light-transmitting member; and a calibration process for determining calibration parameters for calibrating at least the parallax shift caused by the light-transmitting member based on the results of the stereo matching of multiple images obtained by capturing an image of a calibration chart with the multiple cameras in the stereo matching process, wherein the calibration chart includes a pattern, and the length of the pattern in the baseline direction of the multiple cameras is different at least in a first area and a second area, and the calibration parameter is determined based on the results of stereo matching in the first area and the second area of ​​the calibration chart, respectively, assuming that different patterns are the same point.

7. A stereo image calibration method as described in claim 6, characterized in that when the first area is defined as an area on a plane including the lens optical axes of the multiple cameras and including a position on the calibration chart where the perpendicular bisectors of the multiple cameras intersect, the second area has a longer length in the baseline direction of the multiple cameras than the first area.

8. A stereo image calibration method as described in claim 6, characterized in that a third area of ​​the calibration chart is located in a direction perpendicular to the baseline direction of the sensor surface of the camera of the first area of ​​the calibration chart, and the lengths of the baseline direction of the multiple cameras are different between the first area and the third area.

Images