Stereoscopic image processing device and stereoscopic image correction method

Abstract

The invention provides a three-dimensional image processing device and a three-dimensional image correction method, which can correct a wide-angle camera mounted on a vehicle with high precision without increasing the size of a calibration image. The correction calibration map includes a pattern, the lengths of the pattern in the baseline direction of the plurality of cameras differ at least in a first region and a second region, and the correction processing unit finds a correction parameter on the basis of the result of stereo matching by setting the different patterns as the same point in the first region and the second region of the correction calibration map.

Description

Technical Field

[0001] The present invention relates, for example, to a (vehicle-mounted) stereoscopic image processing apparatus and a stereoscopic image correction method mounted on a vehicle. Background Technology

[0002] Stereo cameras are known as devices for three-dimensional object recognition. Stereo cameras utilize the different image capture methods of multiple cameras positioned at different locations, detect the parallax between the multiple cameras based on triangulation, and use this parallax to detect the depth and position of an object, thus accurately detecting and determining the object's location.

[0003] Such stereo cameras are mounted on vehicles such as cars and are used in technologies that detect the location of obstacles (vehicle sensing technology). In vehicle sensing technology, in order to cope with many usage scenarios, it is required to detect obstacles with a wide angle and to detect obstacles at a greater distance (wide-angle and long-angle detection).

[0004] To avoid the effects of dirt and other contaminants, vehicle-mounted stereo cameras are typically installed inside the car's cabin. However, with the increasing wide-angle capabilities, the influence of the windshield cannot be ignored. Previously, to correct mounting misalignment, a calibration process known as alignment was performed during vehicle manufacturing and inspection. But for the sake of wider angles, further adjustments are required.

[0005] Patent Document 1 is known as a technology for correcting stereo cameras that takes into account the influence of the windshield. As a subject of patent document 1, it provides a "stereo camera correction device and correction method that can properly correct stereo cameras using a target plate even in narrow spaces". As a solution, "a correction device for a stereo camera that measures the distance to an object based on parallax includes: a stereo camera 3, which has a pair of cameras 4 and 6 arranged separately with a predetermined baseline length B; a target plate 8, which includes at least two targets 8a and 8b arranged parallel to the arrangement direction of the pair of cameras 4 and 6 and separated by the same distance as the baseline length B; and a processing unit that assumes that the target plate 8 exists at an infinite distance relative to the stereo camera 3, and processes the two targets 8a and 8b contained in the images of the target plate 8 captured by the pair of cameras 4 and 6 respectively as the same target, and corrects the parallax offset of the pair of cameras 4 and 6. In addition, the same correction technique is described in Patent Document 2. As a problem, it is set as "the search range for stereo matching can be variably set according to the position on the image". As a solution, it is set as "having a comparison image line memory 7, an address generation circuit 10 and a stereo matching circuit 8. The line memory 7 stores image data in a reference pixel area in one captured image and image data on a horizontal line corresponding to the vertical position of the reference pixel area in another captured image." Address generation circuit 10 sets the search range for stereo matching and instructs row memory 7 to read image data within the set search range and image data within the reference image region. Furthermore, stereo matching circuit 8 determines the relevant target of the reference pixel region through stereo matching based on the image data within the search range and the image data of the reference pixel region read from row memory 7. Here, the address generation circuit 10 corrects the position of the search range related to the reference pixel region based on the degree of offset of the corresponding point at infinity relative to the horizontal position of the reference pixel region.

[0006] Existing technical documents

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2017-62150

[0009] Patent Document 2: Japanese Patent Application Publication No. 2001-92968 Summary of the Invention

[0010] The problem that the invention aims to solve

[0011] In Patent Documents 1 and 2, a calibration chart is placed near the stereo camera, and the distance between the two cameras of the stereo camera (baseline length) and the period of the pattern on the calibration chart are aligned with the baseline length, virtually placing the calibration chart at infinity. This allows correction of the slope shift of light caused by refraction from the windshield. However, these techniques do not consider the positional shift of light caused by refraction from the windshield, resulting in the problem that complete correction is not possible if the calibration chart is actually placed near the stereo camera.

[0012] The present invention was made in view of the above-mentioned problems, and its object is to provide a stereo image processing apparatus and a stereo image correction method, wherein a calibration map is arranged near the stereo camera, which enables high-precision correction of wide-angle stereo cameras without increasing the size of the calibration map.

[0013] Methods for solving problems

[0014] According to one aspect of the present invention, a stereo image processing apparatus is provided, comprising: a stereo matching unit that performs stereo matching of multiple images captured by multiple cameras to determine parallax, the multiple cameras capturing a subject via a light-transmitting member; and a correction processing unit that, based on the stereo matching result of the stereo matching unit performing stereo matching of multiple images obtained by the multiple cameras capturing a calibration map, determines a correction parameter that at least corrects the parallax shift caused by the light-transmitting member, the calibration map including a pattern, the pattern being a repeating pattern larger than the baseline of the multiple cameras, and the correction processing unit determining the correction parameter based on the result of stereo matching by setting different patterns to the same point.

[0015] Invention Effects

[0016] According to the stereo image processing apparatus and stereo image correction method of the present invention, a stereo image processing apparatus and stereo image correction method capable of high-precision correction of a wide-angle camera without increasing the size of the calibration image can be provided.

[0017] Other issues, structures, and effects not mentioned above will be clarified through the following description of the implementation methods. Attached Figure Description

[0018] Figure 1 This is a block diagram illustrating the structure of the stereo camera image processing apparatus of Embodiment 1.

[0019] Figure 2 This is a diagram illustrating the subject matter of the existing method in Embodiment 1.

[0020] Figure 3 Other figures illustrating the subject matter of the existing method of Embodiment 1.

[0021] Figure 4 The shape of the area shown in the calibration diagram of Example 1 is illustrated.

[0022] Figure 5 The patterns in each region of the calibration diagram of Example 1 are shown.

[0023] Figure 6 The optical path of Embodiment 1 is shown.

[0024] Figure 7 This illustrates the parallax offset of the existing method in Embodiment 1.

[0025] Figure 8 The distance measurement method of the existing method shown in Example 1 is illustrated.

[0026] Figure 9 The parallax offset of Example 1 is shown.

[0027] Figure 10 The distance measurement of Example 1 is shown.

[0028] Figure 11 The interval D of Example 1 is shown.

[0029] Figure 12 Another shape of the area of ​​the calibration map of Example 1 is shown.

[0030] Figure 13 Another shape of the area of ​​the calibration map of Example 1 is shown.

[0031] Figure 14 Another pattern of each region of the calibration diagram of Example 1 is shown.

[0032] Figure 15 This is a block diagram illustrating the structure of the stereo camera image processing apparatus of Embodiment 2 for generating correction data. Detailed Implementation

[0033] Hereinafter, this embodiment will be described with reference to the accompanying drawings. In the drawings, functionally identical elements are sometimes shown with the same numbers. Furthermore, the drawings illustrate embodiments that follow the principles of this disclosure, but they are for the purpose of understanding this disclosure and are in no way intended to limit its interpretation. The descriptions in this specification are merely typical examples and do not limit the scope of patent protection or application of this disclosure in any sense.

[0034] In this embodiment, the description has been sufficiently detailed for the purpose of implementing this disclosure, but other methods are also possible. It should be understood that structural and constructional changes and substitutions of various elements can be made without departing from the scope and spirit of the technical concept of this disclosure. Therefore, the following description is not limited to this interpretation.

[0035] [Example 1]

[0036] Reference Figure 1 The structure of the stereo camera image processing apparatus 10 (hereinafter referred to as "image processing apparatus 10" or "stereo image processing apparatus 10") of Embodiment 1 will be described below. This image processing apparatus 10 is mounted, for example, on a vehicle such as an automobile, and is used to detect the distance from the vehicle to three-dimensional objects (other vehicles, buildings, pedestrians, etc.) around the vehicle. The following description will use the case where the image processing apparatus 10 is mounted on a vehicle as an example, but it is not limited to this.

[0037] Figure 1 This is a block diagram illustrating a structural example of the image processing apparatus 10 of Embodiment 1. The image processing apparatus 10 is configured to detect surrounding three-dimensional objects based on images obtained by the right camera 50 and the left camera 60, and to issue an alarm as needed. The right camera 50 and the left camera 60 constitute a stereo camera. Furthermore, the cameras constituting the stereo camera are not limited to two separate units.

[0038] For example, the image processing device 10 includes an image processing unit 100, a stereo parallax image generation unit 200, a stereo parallax image correction unit 99, a road surface cross-section shape estimation unit 400, a stereoscopic object detection unit 500, and an alarm control unit 700.

[0039] The image processing device 10 generates a stereo disparity image in the stereo disparity image generation unit 200 in an area that can be captured by both the right camera 50 and the left camera 60 (hereinafter referred to as the "stereo vision area"), utilizing the parallax of the right camera 50 and the left camera 60. Furthermore, in the stereo vision stereo object detection unit 500, the distance from the vehicle to the stereo object is measured based on the parallax.

[0040] Although not illustrated, the right camera 50 and left camera 60 are equipped with lenses and image sensors. The right camera 50 and left camera 60 acquire (capture) images of various objects via the lenses and image sensors. In this embodiment, the right camera 50 and left camera 60 are mounted inside the vehicle interior and acquire images of subjects around the vehicle through a light-transmitting component such as a windshield. The image processing device 10 acquires image P1 (first image) from the right camera 50 and image P2 (second image) from the left camera 60.

[0041] As an example, the image processing unit 100 is configured to include affine processing units 20a and 20b, brightness correction units 21a and 21b, pixel interpolation units 22a and 22b, and brightness information generation units 23a and 23b. The image processing unit 100 applies predetermined image processing to images P1 and P2 obtained from the right camera 50 and the left camera 60, and provides them to the stereo parallax image generation unit 200.

[0042] Affine processing unit 20a applies affine processing to image P1 from right camera 50. Affine processing is, for example, a linear coordinate transformation, but may also include nonlinear operations. As a result of this affine processing, affine processing unit 20a obtains image P3 (the third image). Similarly, affine processing unit 20b applies affine processing to image P2 from left camera 60 to obtain image P4 (the fourth image).

[0043] Affine processing units 20a and 20b can also perform distortion transformation processing other than affine processing. In this embodiment, the coordinate system of the fsinθ direction (ftanθx, ftanθy) of the fisheye lens projection method is projected and transformed. Here, f represents the focal length of the fisheye lens, θ represents the angle of view incident on the fisheye lens, and θx and θy represent the horizontal and vertical components of the angle of view incident on the fisheye lens, respectively. Furthermore, in this embodiment, affine processing units 20a and 20b correct the vertical pixel displacement caused by the windshield.

[0044] Brightness correction unit 21a corrects the brightness of each pixel in image P3. For example, it corrects the brightness of each pixel in image P3 based on the gain of the right camera 50, the gain differences of each pixel in image P3, etc. Similarly, brightness correction unit 21b corrects the brightness of each pixel in image P4.

[0045] Pixel interpolation unit 22a performs demosaic processing on image P3. For example, it performs a transformation from a RAW image to a color image. Similarly, pixel interpolation unit 22b performs demosaic processing on image P4.

[0046] The luminance information generation unit 23a generates luminance information for image P3. For example, it transforms information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generation unit 23b generates luminance information for image P4.

[0047] The stereo disparity image generation unit 200 uses the images of the stereo vision region (common field of view region) in the obtained images P3 and P4 to generate a stereo disparity image of the stereo vision region.

[0048] The stereo parallax image generation unit 200 includes an exposure adjustment unit 210 and a sensitivity correction unit 220, capable of performing feedback control to the right camera 50 and left camera 60 regarding exposure, sensitivity, etc. Additionally, the stereo parallax image generation unit 200 also includes a geometry correction unit 230 for geometric correction of the left and right images, and a matching unit 240 for matching the left and right images. The matching unit 240 performs stereo matching of the left and right images to calculate the parallax (stereo parallax image).

[0049] The stereo parallax image correction unit 99 corrects parallax shifts caused by windshields and other light-transmitting components based on data from the stereo parallax correction data recording unit 98 that has been acquired in advance.

[0050] The road surface cross-section shape estimation unit 400 estimates the cross-section shape of a predetermined road surface along which a vehicle equipped with the image processing device 10 will travel.

[0051] The stereoscopic object detection unit 500 detects stereoscopic objects in the stereoscopic vision region based on the stereoscopic parallax image generated by the stereoscopic parallax image generation unit 200. Furthermore, stereo matching is applied to the detected stereoscopic objects to detect parallax, and the category of the stereoscopic object (pedestrian, bicycle, vehicle, building, etc.) is identified. By detecting stereoscopic objects and identifying their categories, categories for preventative safety are further determined. In the case of a vehicle being detected, the detection result can be used for following control of the vehicle ahead, emergency braking control, etc. In the case of a pedestrian or bicycle being detected, emergency braking control and alarm control can be performed. Compared to stationary objects, alarms and controls are implemented for objects flying towards vehicles within a wider field of view. By measuring the distance to these detected objects and estimating the moving speed of objects tracked in a time sequence, the alarm control unit 700 can implement more appropriate alarms and controls.

[0052] exist Figure 1 In this embodiment, the feature is that the data generated by the stereo parallax correction data recording unit 98 is generated.

[0053] Next, the effects of this embodiment will be explained. First, the problems of the prior art will be explained in detail. As described above, in Patent Document 1 (hereinafter referred to as "Prior Art 1") and Patent Document 2 (hereinafter referred to as "Prior Art 2"), a calibration map is arranged near the stereo camera, and the interval (baseline length) between the two cameras of the stereo camera and the period of the pattern of the calibration map are consistent with the baseline length, virtually placing the calibration map at infinity. This allows correction of the effect of the slope shift of light caused by the refraction of the windshield. On the other hand, the effect of the refraction of the windshield is mainly twofold. The first is image distortion caused by the slope shift of light. This image distortion occurs equally in both the nearby calibration map and the distant calibration map. The second is the position shift of light. The position shift of light has a greater impact on the nearby calibration map and the object being measured, and has the characteristic of being able to ignore the distant calibration map and the object being measured.

[0054] Of these two factors, prior art 1 and prior art 2 do not consider the effect of the positional shift of light rays caused by windshield refraction. As mentioned above, this positional shift of light rays is detected when measuring distances to nearby objects, but can be ignored when measuring distant objects. Typically, parallax shift is corrected for optimal distance measurement. On the other hand, if a nearby calibration map is used to correct for parallax shift, a parallax shift occurs when a distant object is detected, resulting in the positional shift of light rays caused by windshield refraction. Therefore, in prior art 1 and prior art 2, if a calibration map is placed nearby, there is a problem that parallax shift cannot be completely corrected.

[0055] Figure 2This is a diagram illustrating the subject matter. This diagram shows a cross-section of the lens pupil position connecting the right camera 50 and the left camera 60. For simplicity, the amount of light displacement caused by the windshield 1 is represented as larger than it actually is. In this diagram, the light rays at a horizontal viewing angle of 0 degrees in the right camera 50 and left camera 60 are designated as light ray R51 and light ray R61, respectively. Furthermore, the interval between light ray R61 and light ray R51 incident on the windshield 1 (as opposed to light ray R61 and light ray R51 relative to the windshield 1 and the stereoscopic camera image processing device 10) is designated as interval D1. Additionally, the light rays on the horizontal wide-angle side in the right camera 50 and left camera 60 are designated as light ray R52 and light ray R62. Similarly, the interval between light ray R62 and light ray R52 incident on the windshield 1 is designated as interval D2. Furthermore, the line extending the light ray R51 between the windshield 1 and the right camera 50 is designated as axis K51, and the line extending the light ray R61 between the windshield 1 and the left camera 60 is designated as axis K61. Similarly, the line extending the ray R52 between the windshield 1 and the right camera 50 is designated as axis K52, and the line extending the ray R62 between the windshield 1 and the left camera 60 is designated as axis K62. Here, the interval between axes K51 and K61, and the interval between axes K52 and K62, are consistent with the baseline length B.

[0056] First, consider a horizontal viewing angle of 0 degrees. Ray R51 is almost identical to axis K51, but not perfectly so. Similarly, ray R61 is identical to axis K61. Therefore, the baseline length B is inconsistent with interval D1. On the other hand, the interval D2 of the horizontal wide-angle is also different from the baseline length B. Here, the characteristic is that intervals D1 and D2 are different due to the influence of the windshield 1. In prior art 1 and prior art 2, the effect of distortion caused by the windshield is estimated by utilizing the case where intervals D1, D2, and baseline length B are consistent. However, in reality, intervals D1, D2, and baseline length B are inconsistent.

[0057] Figure 3 This diagram illustrates the impact of different baseline lengths B and intervals D (intervals D1 and D2). This diagram is also related to... Figure 2Similarly, the cross section connecting the lens pupil positions of the right camera 50 and the left camera 60 is shown. For simplicity, the amount of light displacement caused by the windshield 1 is represented as larger than it actually is. In the calibration diagrams described in prior art 1 and prior art 2, a pattern with a period equal to the baseline length is used. In this diagram, a calibration diagram G10 with the same pattern and a period equal to the baseline length is also provided. In this diagram, the axes of the predetermined viewing angle from the pupil positions of the right camera 50 and the left camera 60 are designated as axes K53 and K63, respectively. Furthermore, the junctions of calibration diagram G10, axes K53, and axes K63 are designated as positions Q1 and Q2, respectively. Since the slopes of axes K53 and K63 are equal, the interval between positions Q1 and Q2 is the baseline length B. This structure is the same as that in prior art 1 and prior art 2. Furthermore, the light rays that detect the images at positions Q1 and Q2 by the right camera 50 and the left camera 60 are designated as light rays R53 and R63, respectively. Furthermore, the extended lines of light rays R53 and R63 between the pupil positions of the right camera 50 and the left camera 60 and the windshield 1 are respectively designated as line S53 and line S63, and the intersection of line S53 and line S63 is designated as position TP1.

[0058] In prior art 1 and prior art 2, virtual detection of infinity is achieved by setting the calibration map G10 as a pattern with a period equal to the baseline length. For this, the two axes, such as axes K53 and K63, must be parallel lines. However, the angle of the light ray R53 incident on the right camera 50 through the windshield 1 is different from the angle of the light ray R63 incident on the left camera 60. Therefore, the stereo camera adjusted in this calibration map produces a large error. In this map, the portion that should be at infinity is detected as the calibration map located at position TP1, making correction by distance L a significant problem.

[0059] Therefore, this is taken into consideration in this embodiment. Figure 4This illustrates the shape of the regions in the calibration map G20 of this embodiment. Furthermore, each region exhibits the same pattern (a quadrilateral shape in the example). In this embodiment, a key feature is that the widths of the horizontal (baseline direction) regions (patterns) of the calibration map G20 differ. For example, it is shown that the horizontal width da1 of region C33 (first region) and the horizontal width da2 of region C35 (second region) of the calibration map G20 are different. Width da2 is larger than width da1. Furthermore, in this figure, when the plane containing the optical axes of the lenses of the two cameras (including the stereo camera) lies on region C33 of calibration map G20, and the perpendicular bisecting lines of the two cameras intersect, (in other words, region C33 is defined as the area on calibration map G20 where the perpendicular bisecting lines of the two cameras intersect, or region 33 is defined as the area on the plane containing the optical axes of the lenses of the two cameras and the plane containing the calibration map G20 and the optical axes of the lenses of the two cameras, and the perpendicular bisecting lines of the two cameras intersect, and region 35 is defined as the area where the horizontal (baseline direction) position differs from region C33), both widths da1 and da2 are greater than the baseline length B, and width da1 is closer to the baseline length B than width da2 (i.e., width da2 is greater than width da1). Additionally, the smaller the radius of curvature of the windshield, the further widths da1 and da2 are from the baseline length B (refer to...). Figure 11 Conversely, the larger the radius of curvature of the windshield, the closer its widths da1 and da2 are to the baseline length B (refer to...). Figure 11 The radius of curvature of the windshield varies depending on the vehicle, therefore a corresponding calibration map interval needs to be set. Furthermore, if the distance between the stereo camera and the calibration map is small, widths da1 and da2 gradually move away from the baseline length B. Conversely, as the distance between the stereo camera and the calibration map increases, widths da1 and da2 approach the baseline length B.

[0060] Figure 5 express Figure 4 The patterns within each region of the calibration map G20 are shown. For example, the pattern within region C33 on the calibration map G20 is shown. Here, the widths of the regions in the horizontal direction are also different. For example, the horizontal width db1 of (0,0) is different from the horizontal width db2 of (2,0). Width db2 is larger than width db1. Furthermore, the detection light intensity of each region is set differently to prevent erroneous matching. By setting the structure in this way, for example, the offset between corresponding points in the image of region C34 of the calibration map G20 detected by the right camera 50 and the image of region C33 detected by the left camera 60 can be detected. As a method for calculating the offset, detection methods such as block matching and feature point extraction used in stereo cameras can be employed.

[0061] Figure 6 The optical path is shown considering the positional shift of light caused by the windshield. First, the light from the left camera 60 is... Figure 3 The same applies. On the other hand, the angle of ray R54 is changed so that the angle of ray R63 incident on the left camera 60 is the same as the angle of ray R54 incident on the right camera 50 (lines S54 and S63 are parallel, and their interval is called the baseline length B). At this time, the periodic pattern in the horizontal direction on the calibration map G20 (in other words, the interval between the positions Q3 and Q2 of the images on the calibration map G20 detected by the right camera 50 and the left camera 60 in ray R54 and ray R63) is arranged with a separation distance D.

[0062] By using with Figure 1 The same processing is used to obtain the parallax shift caused by the slope shift of light due to the refraction of the windshield in this embodiment. That is, in the stereo parallax image correction unit 99, based on the stereo matching result performed by the matching unit 240 on the left and right images (image P1 and image P2) obtained by capturing calibration map G20 using the left and right cameras (right camera 50 and left camera 60), a correction parameter is determined to at least correct the parallax shift caused by the refraction of the windshield (which is a light-transmitting component). At this time, the stereo parallax image correction unit 99 determines the above correction parameter based on the result of stereo matching with different patterns set as the same point. At this time, the result of the correction data of the stereo parallax image correction unit 99 is this parallax shift. The data can be processed to correct this parallax shift, recorded in the stereo parallax correction data recording unit 98, and the stereo parallax image correction unit 99 can perform the correction. If such correction is performed, although high accuracy is achieved for distant measurement objects, parallax shift due to positional shift caused by the refraction of the windshield will occur for nearby measurement objects. However, the tolerance for parallax shift is greater in the vicinity than in the distance, so its impact is small.

[0063] Figure 7 These are simulation results from prior art 1 and prior art 2. The calculations are performed using the following parameters.

[0064] <Windshield>

[0065] radius of curvature

[0066] - Horizontal: 5.0m

[0067] - Vertical: 5.0m

[0068] Windshield tilt angle: 45 degrees

[0069] Windshield thickness: 5.0mm

[0070] Windshield refractive index: 1.52

[0071] 3D camera - distance between windshield: 50mm

[0072] <3D Camera>

[0073] Baseline length: 200mm

[0074] Focal length: 4.0mm

[0075] Sensor pixel pitch: 0.00375mm

[0076] Viewing angle: -72 degrees to +72 degrees (horizontal), 0 degrees (vertical)

[0077] <Calibration Diagram>

[0078] Camera-calibration map distance: 0.5m

[0079] <Object to be measured>

[0080] Stereo camera - for measuring distances between objects: 50m

[0081] (When estimating the impact of the windshield) Stereo camera - measuring the distance between objects: 1000m

[0082] Figure 7 The horizontal viewing angle dependence of parallax shift is shown. The vertical axis represents parallax shift, and the horizontal axis represents the horizontal viewing angle. This figure shows the results with and without a windshield. Furthermore, with a windshield, the difference between the parallax shift detected using a nearby calibration map and the parallax shift when the distance between the stereo camera and the measured object is set to 1000m is calculated. That is, it represents the state after removing the influence of the slope shift of light caused by the refraction of the windshield. Therefore, the parallax shift on the vertical axis is due to the influence of the positional shift of light caused by the refraction of the windshield. In actual correction, since both the slope shift and the positional shift of light caused by the windshield are detected simultaneously, this parallax shift remains if the corrections described in Prior Art 1 and Prior Art 2 are performed.

[0083] Figure 8 The horizontal viewpoint dependence of the ranging distance is shown. The vertical axis represents the ranging distance of the stereo camera, and the horizontal axis represents the horizontal viewpoint. Here, we use... Figure 7 The parallax offset shown is used to calculate the ranging distance of the stereo camera. Accurate ranging (50m) can be achieved without a windshield. However, the ranging error increases with the presence of a windshield. Ranging error also occurs in areas with a small horizontal angle of view (e.g., 0 degrees), but the error increases on the wide-angle side.

[0084] Figure 9This illustrates the horizontal viewing angle dependence of parallax offset when this embodiment is applied. The vertical axis represents parallax offset, and the horizontal axis represents the horizontal viewing angle. Figure 7 Similarly, this is the state after removing the distortion caused by the windshield. Therefore, by applying this embodiment, there is almost no parallax shift. Thus, in this embodiment, the parallax shift of the distortion factor caused by the windshield can be calculated with high accuracy.

[0085] Figure 10 The horizontal viewing angle dependence of the ranging distance when applying this embodiment is shown. The vertical axis represents the ranging distance of the stereo camera, and the horizontal axis represents the horizontal viewing angle. Thus, it can be seen that by applying this embodiment, high-precision ranging (50m) can be achieved.

[0086] Figure 11 The optimal interval D of the calibration plot is represented by Figure 6 (The horizontal viewpoint dependence of the interval of a periodic or repeating pattern). The vertical axis represents the interval D, and the horizontal axis represents the horizontal viewpoint. Figure 9 , Figure 10 The parallax shift and the ranging distance of the stereo camera are calculated using this interval D. For example... Figure 11 As shown, intervals D longer than the baseline (200 mm) are set in all horizontal angular regions. Thus, we obtain... Figure 9 , Figure 10 The results shown are of high precision.

[0087] As described above, this embodiment, like prior art 1 and prior art 2, compares different regions (assuming different patterns are the same point for stereo matching) and detects their offset. The difference between this embodiment and prior art 1 and prior art 2 is that, as... Figure 6 As shown, the distance D of the periodic pattern in the horizontal direction on calibration map G20 is different from the baseline length B (more specifically, it is larger than the baseline length B). Moreover, this embodiment can improve ranging accuracy by making the horizontal lengths of the two cameras different at least in the first and second regions.

[0088] Here, in this embodiment, as Figure 4 As shown, the vertical viewing angle is the same, but for example, Figure 12As shown, the amount of change in the width in the horizontal direction (baseline direction) can also be altered relative to the vertical viewing angle. For example, the width in the horizontal direction (baseline direction) can be changed in region C33 (first region) of calibration map G20 and in regions C13, C23, C43, and C53 (third region) of region C33 (first region) that are perpendicular to the baseline direction of the camera's sensor surface. This is because the windshield is tilted, so the optimal interval D in the horizontal direction (baseline direction) changes relative to the vertical viewing angle. This improves the overall ranging accuracy of the image. Furthermore, it is possible to... Figure 12 Connecting the various areas in such a phased (step-like) manner can also be like... Figure 13 That way, the areas are connected smoothly. Moreover, for example, even using two intervals D, one for the narrow horizontal corner and one for the wide horizontal corner, the desired effect can still be achieved.

[0089] Moreover, in this embodiment, the following was used Figure 5 Such a quadrilateral shape calibration pattern, but it can also be like... Figure 14 That way, a circular calibration pattern is used.

[0090] Furthermore, despite Figure 1 The stereo parallax image correction unit 99 and the stereo parallax correction data recording unit 98 are used to correct the effects of the windshield, but the present invention is not limited thereto. For example, the correction data from the stereo parallax correction data recording unit 98 can be sent to the affine processing unit 20a or the affine processing unit 20b for correction.

[0091] [Example 2]

[0092] Reference Figure 15 This section describes the correction method of the image processing apparatus 10 in Embodiment 2. The correction method in Embodiment 1 uses the same calculation method as the correction methods in Prior Art 1 and Prior Art 2, but is not limited thereto. In this Embodiment 2, a correction method is described that takes into account the positional shift of light rays refracted by the windshield when the calibration map is arranged in a nearby configuration.

[0093] Figure 15 An example of the structure of an image processing apparatus 10, including an estimation process for the effects of a windshield, is shown. In this embodiment, similar to Embodiment 1, it is characterized by using... Figure 4 , Figure 12 , Figure 13 The calibration diagram is shown. In this embodiment, as... Figure 15As shown, the Y-offset image generation unit 201 generates vertical offset (Y-offset) images of the two images. Then, the ΔY-offset image generation unit 80 calculates the difference (ΔY-offset image) between the Y-offset image and the Y-offset image under reference conditions obtained from the actual machine. Then, the Δparallax offset calculation processing unit 85 generates a Δparallax offset image by multiplying the ΔY-offset image by a predetermined coefficient. Finally, the parallax offset calculation processing unit 90 adds the Δparallax offset image to the parallax offset image under reference conditions, thereby calculating the parallax offset due to the slope offset factor of light caused by the refraction of the windshield. Here, Δparallax offset and ΔY-offset represent the parallax offset and Y-displacement amount under reference conditions such as design value and product center value. Here, Δparallax offset and ΔY-offset are used to address fluctuations in the radius of curvature of the windshield, the mounting position of the camera relative to the windshield, and thickness shifts of the windshield.

[0094] Next, we will explain why parallax offset can be corrected in this embodiment. The windshield has horizontal and vertical radii of curvature and a tilt angle. Therefore, factors such as fluctuations in the windshield's radius of curvature, the camera's mounting position relative to the windshield, and thickness variations in the windshield cause not only horizontal changes (Δparallax offset) but also vertical changes (ΔY offset). Here, Δparallax offset is related to ΔY displacement. Therefore, Δparallax offset can be calculated by detecting ΔY offset. Then, the parallax offset can be calculated from the Δparallax offset and the parallax offset under reference conditions. In this embodiment, the parallax offset data calculated in this way is recorded in... Figure 1 The stereo parallax correction data recording unit 98 shown is used, and the correction is performed by the stereo parallax image correction unit 99.

[0095] As described above, by considering the positional shift of light rays associated with the refraction of the windshield when the calibration map is positioned near it, the effect of the slope shift of light rays caused by the refraction of the windshield can be estimated with high accuracy. Here, in this embodiment, the offset in the Y direction is used, but it is not limited to this; any method using the calibration map to correct for the windshield's influence can achieve the same effect. Furthermore, when the influence of fluctuating factors such as the windshield and camera mounting position is small, the correction method of Embodiment 1 can be used. On the other hand, when the influence of these fluctuating factors is large, the correction method of this embodiment is preferred.

[0096] [Summarize]

[0097] As described above, the stereoscopic image processing apparatus 10 of this embodiment includes: a stereo matching unit (matching unit 240) that performs stereo matching of multiple images captured by multiple cameras (right camera 50, left camera 60) to determine parallax, wherein the multiple cameras capture a subject via a light-transmitting member (windshield 1); and a correction processing unit (stereoscopic parallax image correction unit 99) that, based on the stereo matching result of the stereo matching unit (matching unit 240) performing stereo matching of multiple images obtained by the multiple cameras capturing a correction calibration map, determines a correction parameter that at least corrects the parallax shift caused by the stereo matching unit (matching unit 240), wherein the correction calibration map includes a pattern, the pattern being a repeating pattern larger than the baseline of the multiple cameras (…). Figure 11 The correction processing unit (stereo parallax image correction unit 99) calculates the correction parameters based on the result of stereo matching by setting different patterns to the same point.

[0098] Furthermore, the stereoscopic image processing apparatus 10 of this embodiment includes: a stereo matching unit (matching unit 240) that performs stereo matching of multiple images captured by multiple cameras (right camera 50, left camera 60) and calculates the parallax, wherein the multiple cameras capture a subject through a light-transmitting component (windshield 1); and a correction processing unit (stereoscopic parallax image correction unit 99) that, based on the stereo matching result of the multiple images obtained by the stereo matching unit (matching unit 240) from the correction calibration map captured by the multiple cameras, calculates a correction parameter that at least corrects the parallax shift caused by the light-transmitting component (windshield 1), wherein the correction calibration map includes a pattern whose length in the baseline direction of the multiple cameras is different in at least a first region and a second region (in other words, having a first region and a second region with different lengths in the baseline direction of the multiple cameras). Figure 4 da2 > da1, Figure 11 The correction processing unit (stereo parallax image correction unit 99) calculates the correction parameters based on the result of stereo matching by setting different patterns to the same point in the first region and the second region of the correction calibration map, respectively.

[0099] Furthermore, in this embodiment, when the stereoscopic image processing apparatus 10 defines the region on the calibration map containing the optical axes of the lenses of the plurality of cameras and the position on the calibration map where the perpendicular bisecting lines of the plurality of cameras intersect as the first region (and the region where the position of the baseline direction of the plurality of cameras differs from the first region as the second region), the length of the baseline direction of the plurality of cameras increases relative to the first region in the second region (in other words, as the position of the baseline direction of the plurality of cameras moves away from the first region, the length of the baseline direction of the plurality of cameras increases relative to the first region). Figure 4 da2 > da1 Figure 11 wait).

[0100] Furthermore, in this embodiment, the stereo image processing apparatus 10 has a third region of the calibration map in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first region of the calibration map, and the lengths of the plurality of cameras in the baseline direction are different in the first region and the third region. Figure 12 , Figure 13 ).

[0101] The stereo image correction method according to this embodiment includes: a stereo matching process (matching unit 240) that performs stereo matching of multiple images captured by multiple cameras (right camera 50, left camera 60) to determine parallax, wherein the multiple cameras capture a subject via a light-transmitting component (windshield 1); and a correction process (stereo parallax image correction unit 99) that, based on the stereo matching result of multiple images obtained by capturing a correction calibration map by the multiple cameras in the stereo matching process (matching unit 240), determines a correction parameter that at least corrects the parallax shift caused by the light-transmitting component (windshield 1), wherein the correction calibration map includes a pattern, the pattern being a repeating pattern larger than the baseline of the multiple cameras. Figure 11 In the correction process (stereo parallax image correction unit 99), the correction parameters are determined based on the result of stereo matching by setting different patterns to the same point.

[0102] Furthermore, the stereo image correction method of this embodiment includes: a stereo matching process (matching unit 240) that performs stereo matching of multiple images captured by multiple cameras (right camera 50, left camera 60) to determine parallax, wherein the multiple cameras capture a subject via a light-transmitting component (windshield 1); and a correction process (stereo parallax image correction unit 99) that, based on the stereo matching result of multiple images obtained by capturing a correction calibration map by the multiple cameras in the stereo matching process (matching unit 240), determines a correction parameter that at least corrects the parallax shift caused by the light-transmitting component (windshield 1), wherein the correction calibration map includes a pattern whose length in the baseline direction of the multiple cameras is different in at least a first region and a second region (in other words, having a first region and a second region with different lengths in the baseline direction of the multiple cameras). Figure 4 da2 > da1, Figure 11 In the correction process (stereo parallax image correction unit 99), the correction parameters are determined based on the result of stereo matching by setting different patterns to the same point in the first region and the second region of the correction calibration map, respectively.

[0103] That is, the stereoscopic image processing apparatus 10 and the stereoscopic image correction method of this embodiment change the pattern size of the calibration map (correction calibration map) G20 used for correction from the center to the outside, thereby suppressing the positional shift caused by the refraction of light-transmitting components such as windshields.

[0104] According to the stereo image processing apparatus 10 and stereo image correction method of this embodiment, a stereo image processing apparatus 10 and stereo image correction method that can correct a wide-angle camera with high precision without increasing the size of the calibration map can be provided.

[0105] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the specific structure is not limited to these embodiments. Even if there are design changes that do not depart from the spirit of the present invention, they are also included in the present invention.

[0106] Furthermore, the present invention is not limited to the embodiments described above, but includes various modifications beyond those described. For example, the embodiments described above are examples that have been explained in detail for the purpose of easily understanding the present invention, and are not necessarily limited to having all the structures described.

[0107] Furthermore, the aforementioned structures, functions, processing units, and processing modules can be partially or entirely implemented in hardware, for example, through integrated circuit design. Alternatively, the aforementioned structures and functions can be implemented in software by a processor interpreting and executing programs that implement each function. The programs, tables, files, and other information implementing these functions can be stored in storage devices such as memory, hard disks, and SSDs (Solid State Drives), or on recording media such as IC cards, SD cards, and DVDs.

[0108] Furthermore, control lines and information lines represent the lines that are considered necessary in the instructions, but may not represent all control lines and information lines on the product. In fact, it can be considered that almost all structures are interconnected.

[0109] Symbol Explanation

[0110] 1 windshield

[0111] 10. Stereo camera image processing device (stereo image processing device)

[0112] 50 right camera

[0113] 60 Left Camera

[0114] 20a Affine Processing Unit

[0115] 20b Affine Processing Unit

[0116] 98 Stereo Parallax Correction Data Recording Department

[0117] 99. Stereo parallax image generation unit (correction processing unit)

[0118] 200 stereo parallax image generation unit

[0119] 201 Y-offset image generation unit

[0120] 400 Road Surface Cross-Section Shape Estimation Section

[0121] 500 Stereo Vision Stereo Object Inspection Department.

Claims

1. A stereoscopic image processing device, characterized in that, have: A stereo matching unit performs stereo matching of multiple images captured by multiple cameras to determine parallax, wherein the multiple cameras capture images of the subject via light-transmitting components; and The correction processing unit, based on the stereo matching result of the stereo matching unit performing stereo matching on multiple images obtained from the calibration maps captured by the multiple cameras, determines correction parameters that at least correct the parallax shift caused by the light-transmitting component. The calibration diagram contains a pattern. The pattern is a repeating pattern that is larger than the baseline of the plurality of cameras. The correction processing unit calculates the correction parameters based on the results of stereo matching by setting different patterns to the same point.

2. A stereoscopic image processing device, characterized in that, have: A stereo matching unit performs stereo matching of multiple images captured by multiple cameras to determine parallax, wherein the multiple cameras capture images of the subject via light-transmitting components; and The correction processing unit, based on the stereo matching result of the stereo matching unit performing stereo matching on multiple images obtained from the calibration maps captured by the multiple cameras, determines correction parameters that at least correct the parallax shift caused by the light-transmitting component. The calibration diagram contains a pattern. The length of the pattern in the baseline direction of the plurality of cameras is different at least in the first and second regions. The correction processing unit calculates the correction parameters based on the result of stereo matching by setting different patterns to the same point in the first and second regions of the correction calibration map.

3. The stereoscopic image processing apparatus according to claim 2, characterized in that, When the region on the calibration map containing the optical axes of the lenses of the plurality of cameras and the location of the position on the plane containing the intersection of the vertical bisecting lines of the plurality of cameras is defined as the first region, the second region is longer in the baseline direction of the plurality of cameras relative to the first region.

4. The stereoscopic image processing apparatus according to claim 2, characterized in that, A third region of the calibration map is located in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first region of the calibration map. The lengths of the plurality of cameras in the baseline direction differ in the first region and the third region.

5. A method for stereoscopic image correction, characterized in that, have: Stereo matching processing involves performing stereo matching on multiple images captured by multiple cameras to determine the parallax, wherein the multiple cameras capture images of the subject through light-transmitting components; and The correction process, based on the stereo matching results of multiple images obtained from the calibration maps captured by the multiple cameras in the stereo matching process, determines correction parameters that at least correct the parallax shift caused by the light-transmitting component. The calibration diagram contains a pattern. The pattern is a repeating pattern that is larger than the baseline of the plurality of cameras. In the correction process, the correction parameters are determined based on the results of stereo matching by setting different patterns to the same point.

6. A method for stereoscopic image correction, characterized in that, have: Stereo matching processing involves performing stereo matching on multiple images captured by multiple cameras to determine the parallax, wherein the multiple cameras capture images of the subject through light-transmitting components; and The correction process, based on the stereo matching results of multiple images obtained from the calibration maps captured by the multiple cameras in the stereo matching process, determines correction parameters that at least correct the parallax shift caused by the light-transmitting component. The calibration diagram contains a pattern. The length of the pattern in the baseline direction of the plurality of cameras is different at least in the first and second regions. In the correction process, the correction parameters are determined based on the results of stereo matching by setting different patterns to the same point in the first and second regions of the correction calibration map, respectively.

7. The stereoscopic image correction method according to claim 6, characterized in that, When the region on the calibration map containing the optical axes of the lenses of the plurality of cameras and the location of the position on the plane containing the intersection of the vertical bisecting lines of the plurality of cameras is defined as the first region, the second region is longer in the baseline direction of the plurality of cameras relative to the first region.

8. The stereoscopic image correction method according to claim 6, characterized in that, A third region of the calibration map is located in a direction perpendicular to the baseline direction of the sensor surface of the camera in the first region of the calibration map. The lengths of the plurality of cameras in the baseline direction differ in the first region and the third region.

Images