Image Processing Device

Abstract

To provide an image processing device capable of accurately distinguishing between a control target such as a fallen object on a road surface and road surface paint and the like and capable of accurately detecting the control target having a low level difference, through high-precision correction of parallax errors resulting from vertical displacement between left and right cameras.SOLUTION: An image processing device includes: a specific region extracting unit for extracting a specific region on a road surface from a plurality of images captured by each of a plurality of vehicle-mounted cameras; a road surface parameter estimating unit for estimating a road surface structure represented by a road surface parameter, from a road surface part of the images, being a road surface other than the specific region; a specific region parallax estimating unit for obtaining ideal parallax that should be obtained in the specific region, from the road surface structure; a parallax image generating unit for obtaining parallax from the plurality of images; and a vertical displacement calculating unit for calculating an amount of vertical displacement of the plurality of images, using the actual parallax of the specific region, the ideal parallax in the specific region, and arrangement information of the specific region in the images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method for recognizing an environment ahead of a vehicle based on a pair of images of the vehicle ahead captured by an on-board camera. [Background technology]

[0002] Some recent automobiles are equipped with a system that recognizes moving objects such as preceding vehicles and pedestrians, obstacles such as guardrails and fallen objects on the road, and road paint such as white lines based on captured images of the area in front of the vehicle, and issues an alarm to the driver if there is a possibility of contact with a moving object or obstacle, or automatically brakes and steers the vehicle to avoid contact, or automatically steers the vehicle to follow the white lines. An image processing device that uses a stereo camera is known as an image processing device used in this type of system, as disclosed in Patent Document 1.

[0003] The abstract of Patent Document 1 states, "A three-dimensional object is accurately detected from stereo images even when there is a vertical shift in the stereo images," and claim 2 of the same document states, "A feature point where both a vertical edge and a horizontal edge exist in the first region is detected, a corresponding point corresponding to the feature point is detected in the second region, and the vertical shift between the feature point in the first region and the corresponding point in the second region is detected as the vertical shift between the first region and the second region." In other words, the image processing device of Patent Document 1 accurately detects a three-dimensional object by detecting a corresponding point in the second image that corresponds to the feature point in the first image of the stereo images, detecting the vertical shift between the first image and the second image from the vertical shift between the feature point and the corresponding point, and calculating the parallax taking the vertical shift into account. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-095646 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the vertical deviation detection method of Patent Document 1 requires identifying corresponding points in the second image that correspond to feature points in the first image. Therefore, in an environment where there are few features in the stereo images and it is difficult to detect feature points or corresponding points (for example, when only a uniform road surface is captured), or in an environment where there are many candidate corresponding points in the second image that correspond to feature points in the first image and it is difficult to identify true corresponding points (for example, when a guidance zone (so-called zebra zone), which is a type of road paint), it is difficult to detect vertical deviations between stereo images, and it may not be possible to accurately distinguish between low-level obstacles (such as objects fallen on the road or animal carcasses) and road paint or shadows, and objects fallen on the road may be mistaken for road paint or shadows, or road paint or shadows may be mistaken for obstacles.

[0006] Therefore, the present invention aims to provide an image processing device that can accurately distinguish between control objects such as objects falling on the road surface and road paint, etc., by highly accurately correcting the parallax error caused by the vertical misalignment between the left and right cameras, and can accurately detect control objects with low steps. [Means for solving the problem]

[0007] In order to solve the above problem, the image processing device of the present invention has a specific area extraction unit that extracts a specific area on the road surface from multiple images captured by each of multiple vehicle-mounted cameras; a road surface parameter estimation unit that estimates the road surface structure represented by road surface parameters from the road surface portion of the image other than the specific area; a specific area disparity estimation unit that calculates the ideal disparity that should be obtained in the specific area from the road surface structure; a disparity image generation unit that calculates the disparity from the multiple images; and a vertical shift calculation unit that calculates the vertical shift amount of the multiple images using the actual disparity of the specific area, the ideal disparity in the specific area, and arrangement information in the image of the specific area. [Effects of the Invention]

[0008] According to the image processing device of the present invention, by correcting the parallax error caused by the vertical misalignment between the left and right cameras with high precision, it is possible to accurately distinguish between control objects such as objects falling on the road surface and road paint, etc., and to accurately detect control objects with low steps. [Brief explanation of the drawings]

[0009] [Figure 1] Image Processing Device [Figure 2] Explanation of vertical deviation [Figure 3] Vertical misalignment correction method [Figure 4] Explains how road surface parameters are estimated [Figure 5A] Explaining the ideal disparity estimation method [Figure 5B] An example of an image captured under the environment of FIG. 5A [Figure 6] Processing flow for generating parallax images [Figure 7] Processing flow of the vertical deviation calculation unit DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of an image processing device and an image processing method of the present invention will be described with reference to the drawings, which recognize the environment ahead of a vehicle based on a pair of images of the vehicle ahead taken by an on-board camera.

[0011] (Overall configuration) Fig. 1 shows an overall configuration diagram of an image processing device 100 according to this embodiment. The image processing device 100 according to this embodiment is an in-vehicle device used to recognize the environment ahead of the vehicle, and as shown in Fig. 1, includes a sensor unit 1, a memory unit 2, a parallax image generation unit 3, a specific area extraction unit 4, an ideal parallax estimation unit 5, a vertical deviation amount calculation unit 6, and a parallax image analysis unit 7. The analysis results from the parallax image analysis unit 7 are input to an ECU (Electronic Control Unit), not shown, and are used to realize an autonomous driving system or a driving assistance system.

[0012] 1, the components other than the sensor unit 1 are specifically a computer including hardware such as a calculation unit such as a CPU, a storage device such as a semiconductor memory, and a communication device. The calculation unit executes a predetermined program to realize the functions of the parallax image generation unit 3, the specific area extraction unit 4, and the like. Below, the details of each unit will be described while omitting such well-known techniques in the computer field as appropriate.

[0013] <Sensor part 1> The sensor unit 1 is an on-board camera installed facing a predetermined direction around the vehicle, for example, toward the front of the vehicle, and is composed of two imaging devices: a left camera 11 and a right camera 12. Note that, hereinafter, this embodiment will be described assuming that the sensor unit 1 is a stereo camera, but a camera system composed of two monocular cameras spaced a predetermined distance apart may also be used as the sensor unit 1.

[0014] <Storage section 2> The storage unit 2 is a storage device such as a semiconductor memory, and is composed of a captured image storage unit 21, a parallax image storage unit 22, a geometric correction information storage unit 23, and a vertical deviation amount storage unit 24.

[0015] The captured image storage unit 21 stores the captured images of the left camera 11 and the right camera 12.

[0016] The parallax image storage unit 22 stores the parallax image P generated by the parallax image generating unit 3. D Remember.

[0017] The geometric correction information storage unit 23 stores geometric correction information for the images captured by the left camera 11 and the right camera 12. This geometric correction information is information for correcting distortions and positional deviations of the captured images caused by lens distortions and optical axis deviations of the left camera 11 and the right camera 12, and is information that is set in advance at the time of shipping from the factory in accordance with the optical characteristics and the like of the sensor unit 1. Note that the images captured by the left and right cameras that have been geometrically corrected using the geometric correction information will be referred to as left image P below. L , Right image P R It is called.

[0018] The vertical deviation amount storage unit 24 stores the vertical deviation amount dj between the left camera 11 and the right camera 12 that occurs due to aging or the like.

[0019] Here, the influence of vertical misalignment in a typical stereo camera will be explained using FIG. 2. In a typical stereo camera, the left image P L and the right image P R By identifying the common imaging area, the parallax is calculated and 3D information of the area in front of the camera is obtained.

[0020] In the stereo camera shown in Fig. 2(a), where there is no vertical misalignment between the left and right images, the left image P L a predetermined area (for example, a first area R1 including a part of a diagonal line such as a white line or a shadow) in the right image P R Since the second region R2 in the right image P is at the same vertical position (first vertical position Y1), R By searching horizontally for the first vertical position Y1 above, the second region R2, which is the common imaging region of the first region R1, can be easily identified, and accurate three-dimensional information (in this example, three-dimensional information indicating that the diagonal lines are not a three-dimensional object) can be obtained from accurate parallax.

[0021] On the other hand, in the stereo camera where the vertical misalignment of the left and right images occurs as shown in FIG. 2(b), the right image P R In this case, a diagonal line with the same inclination as that of the first region R1 is also captured in the third region R3 within the image, and therefore the first region R1 and the third region R3 may be erroneously matched. In this case, a parallax shift amount dD [pix] occurs between the second region R2, which is the original common imaging region, and the third region R3, which is not the original common imaging region.

[0022] Here, if the slope of the diagonal lines in the left and right images is θ [rad] and the vertical shift amount of the left and right images is dj [pix], the parallax shift amount dD is calculated using Equation 1. Furthermore, if the parallax value is D, the height shift amount dY of the target object when the parallax shift amount dD occurs can be calculated using Equation 2 relative to the camera height Y.

[0023]

number

[0024]

number

[0025] In other words, a stereo camera with vertical misalignment calculates a height misalignment dY using Equation 2 that does not actually exist, resulting in the problem of erroneously detecting diagonal lines such as white lines or shadows on the road surface as three-dimensional objects.Since vertical misalignment in a stereo camera inevitably occurs due to factors such as aging and temperature changes in the imaging device, it is necessary to constantly estimate and correct the vertical misalignment while the stereo camera is in operation.

[0026] Therefore, the vertical deviation amount storage unit 24 of this embodiment stores the vertical deviation amount estimated by the vertical deviation amount calculation unit 6 as a prerequisite for processing to correct the vertical deviation. The vertical deviation amount stored in the vertical deviation amount storage unit 24 is used when the parallax image generation unit 3 calculates the parallax, thereby correcting the vertical deviation between the imaging devices and accurately estimating the parallax. As a result, it is possible to prevent erroneous detection of diagonal lines such as white lines or shadows on the road surface as three-dimensional objects. The vertical deviation amount storage unit 24 may store only the vertical deviation amount output by the vertical deviation amount calculation unit 6 for the current frame, or may store multiple vertical deviation amounts output in the past by the vertical deviation amount calculation unit 6 as time-series information.

[0027] <Parallax image generator 3> The parallax image generating unit 3 is composed of a geometric correction unit 31 and a parallax calculation unit 32, and receives as input the left and right images stored in the captured image storage unit 21, the geometric correction information stored in the geometric correction information storage unit 23, and the vertical displacement amount dj stored in the vertical displacement amount storage unit 24, and generates a parallax image P D and outputs it to the parallax image storage unit 22 in the storage unit 2.

[0028] The geometric correction unit 31 receives the left and right images stored in the captured image storage unit 21 and the geometric correction information stored in the geometric correction information storage unit 23 as inputs and performs geometric correction on the captured image. This corrects distortion and positional deviation of the captured image caused by lens distortion and optical axis deviation. Furthermore, if a vertical deviation amount dj is stored in the vertical deviation amount storage unit 24, the geometric correction unit 31 also uses that value for the geometric correction.

[0029] Here, a method for correcting vertical deviation using the vertical deviation amount dj will be described with reference to FIG. 3. As shown in the figure, the left image P L and the right image P R When the vertical deviation amount of the right image P is dj, the geometric correction unit 31 refers to the vertical deviation amount dj [pix] stored in the vertical deviation amount storage unit 24 and calculates the vertical deviation amount of the right image P R Corrected right image P R At this time, if the vertical deviation amount storage unit 24 has stored a plurality of past vertical deviation amounts, one plausible value may be calculated from the plurality of past vertical deviation amounts and set as dj[pix]. As a method for determining one value from a plurality of vertical deviation amounts, a statistical method may be used to remove outliers and calculate the average. Alternatively, any other method may be used for the determination. In FIG. 3, the right image P R By moving vertically, the corrected right image P R ' is generated, but the left image P L By moving vertically, the corrected left image P L ' may be generated.

[0030] The parallax calculation unit 32 calculates a parallax image P from the left and right images corrected by the geometric correction unit 31. D Specifically, a template image of a predetermined size is extracted from one captured image, and the difference (parallax) in the captured position of the same object is estimated by template matching with the other captured image. By calculating the parallax for the entire region of the captured image, a parallax image P is generated in which the parallax is stored in each pixel. D Generate.

[0031] <Specific area extraction part 4> The specific region extraction unit 4 extracts a specific region from the captured image stored in the captured image storage unit 21. In this embodiment, the specific region refers to a diagonal line region on the road surface (see FIG. 2 ), and includes various linear regions such as white lines, road paint, shadows, and road dirt regions. For example, the white line region is extracted by detecting white edges contained in the captured image. This method is not limited to this, and various linear regions such as white lines, road paint, shadows, and road dirt regions may be estimated using a pixel-level type estimation method such as semantic segmentation. When the white line region is extracted, a white line region map in which each pixel is classified as a white line or not by a value of 0 or 1 is output to the ideal parallax estimation unit 5 and the vertical deviation calculation unit 6. Furthermore, when a pixel-level type estimation method such as semantic segmentation is used, a type map in which type information (white lines, road paint, shadows, road dirt, etc.) is stored for each pixel may be output. In addition, if the region is estimated by modeling (straight lines, broken lines, etc.), the parameters of the model may be output. Note that AI technology such as deep learning may be used to improve the accuracy of extracting specific regions.

[0032] <Ideal parallax estimation unit 5> The ideal parallax estimation unit 5 estimates the ideal parallax of each part in the captured image, and is composed of a road surface parameter estimation unit 51 and a specific area parallax estimation unit 52.

[0033] The road surface parameter estimation unit 51 estimates the parallax image P stored in the parallax image storage unit 22. D and the output of the specific region extraction unit 4, and estimates the road surface structure ahead of the vehicle. The method for estimating road surface parameters will be explained below with reference to FIG.

[0034] First, a parallax image P shown in FIG. 4(b) is generated from the captured image shown in FIG. 4(a) (although not shown, it is actually a pair of captured images). D In the road surface area of ​​Figure 4(b), the disparity values ​​stored in each pixel change smoothly as the vertical position of the image changes. On the other hand, it can be seen that three-dimensional objects such as vehicles are located at the same distance from the vehicle and have the same disparity values.

[0035] Parallax image P in Fig. 4(b) D 4(c) is created, three-dimensional objects such as vehicles are represented by vertical lines, and road surface areas are represented by straight lines sloping downward to the right. Therefore, the road surface parameter estimation unit 51 can estimate the road surface structure by determining the parameters of these straight lines sloping downward to the right.

[0036] When creating a v-disparity map, if the disparity of the specific region estimated by the specific region disparity estimation unit 52 is also plotted, the straight line parameters may not be accurately estimated due to the inclusion of a large amount of erroneous disparity. For this reason, it may be possible to add a noise removal process so that the disparity of the specific region estimated by the specific region disparity estimation unit 52 is not plotted on the v-disparity map. Furthermore, the present invention is not limited to this, and any road surface structure estimation method may be used.

[0037] The road surface parameter estimation unit 51 may also output the reliability of the parameter estimation results. Because the vertical deviation calculation unit 6 calculates the vertical deviation based on the road surface parameters estimated by the road surface parameter estimation unit 51, it is conceivable to determine whether the current driving environment is suitable for calculating the vertical deviation based on the reliability of the parameter estimation results. The reliability of the parameter estimation results can be calculated using the amount of disparity used to create the v-disparity map. The amount of information plotted on the v-disparity map decreases due to light intensity phenomena caused by road surface patterns or building shadows, or in environments where road surface disparity is difficult to obtain, such as at night. Since accuracy decreases when there is little information used to calculate the line parameters, it is conceivable to use the number of disparity information plotted on the v-disparity map as the reliability of the parameter estimation results. Alternatively, the degree of fitting when fitting the line parameters to the v-disparity map may be used as the reliability. It is also conceivable to refer to the road surface parameter estimation results in chronological order and set a high reliability if a flat road surface is continuously observed. Furthermore, if weather and time information can be acquired, the reliability may be lowered when it is difficult to calculate the road surface disparity (nighttime, rainy weather, etc.).

[0038] The specific area disparity estimation unit 52 estimates the ideal disparity of the specific area (hatched) from the road surface parameters output by the road surface parameter estimation unit 51. The method of estimating the ideal disparity will be described with reference to Figs. 5A and 5B.

[0039] Figure 5A shows a Cartesian coordinate system with the installation position of the stereo camera on the vehicle going uphill as the origin, the horizontal axis z representing the depth direction, and the vertical axis y representing the vertical direction. In this Cartesian coordinate system, the road surface is represented by a straight line shown in Equation 3. Here, a in Equation 3 is the slope of the road surface when the center of the camera is the origin, and b is the intercept of the road surface when the center of the camera is the origin.

[0040]

number

[0041] FIG. 5B shows a left image P captured by the stereo camera of FIG. 5A. L This is an example of the right image P R (The image is also captured), with the center of the image as the origin, the horizontal axis x indicates the horizontal direction of the image, and the vertical axis y indicates the vertical direction of the image. In this case, the ideal disparity D of the white line area (dashed frame in Figure 5B) at y in the image coordinates is calculated using Equation 4. In Equation 4, B is the baseline length of the stereo camera, f is the focal length, and c is the cell size.

[0042]

number

[0043] The specific area disparity estimation unit 52 performs the calculation of Equation 4 for all specific areas estimated by the specific area extraction unit 4 to generate an ideal disparity map storing the ideal disparity for each specific area, and outputs the ideal disparity map to the vertical deviation calculation unit 6. Furthermore, from the viewpoint of reducing processing costs, the ideal disparity may be calculated and output for only some of the multiple specific areas.

[0044] <Vertical deviation amount calculation unit 6> The vertical deviation amount calculation unit 6 calculates the vertical deviation amount dj between the left and right images captured by the stereo camera based on the specific area information extracted by the specific area extraction unit 4 and the ideal parallax information estimated by the ideal parallax estimation unit 5, and is composed of a specific area parallax acquisition unit 61, an ideal parallax acquisition unit 62, a processing area determination unit 63, a placement information acquisition unit 64, a correction feasibility determination unit 65, and a correction amount calculation unit 66.

[0045] The specific area parallax acquisition unit 61 receives the specific area information (the white line area map, type map, model parameters, etc.) output by the specific area extraction unit 4, and acquires parallax information of the corresponding area from the parallax image storage unit 22.

[0046] The ideal parallax acquisition unit 62 acquires the ideal parallax map estimated by the ideal parallax estimation unit 5 .

[0047] The processing area determination unit 63 determines the processing area for which the correction amount calculation unit 66 calculates the vertical deviation. If the specific area is large, the number of images for which the vertical deviation is to be calculated increases. Therefore, from the perspective of reducing processing costs, it is possible to limit the specific area for which the vertical deviation is calculated. If the specific area parallax acquisition unit 61 acquires a type map in which type information (white lines, road paint, shadows, road dirt, etc.) is stored for each pixel, the processing area is limited by selecting the type to be used. For example, since white lines and road paint, which are displayed in conspicuous white on the road surface, are easy to extract, it is possible to set only the specific area labeled as white lines or road paint as the processing area. Furthermore, considering that the accuracy of parallax calculation decreases with increasing distance from the image capture device, the processing area may be set by selecting a predetermined number of data items in ascending order of proximity to the image capture device. Alternatively, the processing area may be set based on any other criteria.

[0048] The layout information acquisition unit 64 calculates layout information of the specific area using the specific area information (the white line area map, type map, model parameters, etc.) output by the specific area extraction unit 4. The layout information of the specific area is the angle of the diagonal line on the captured image, for example, the slope θ of the diagonal line in FIG.

[0049] The correction possibility determination unit 65 determines whether the current driving environment is suitable for calculating the vertical deviation amount dj, and causes the correction amount calculation unit 66 to calculate the correction amount only when it is determined that the environment is suitable. For example, when the road surface parameter estimation unit 51 outputs the reliability of the road surface parameter estimation result, if the value exceeds a predetermined threshold, it is determined that the calculation is possible.

[0050] If the correction possibility determination unit 65 determines that the current driving environment is suitable for calculating the vertical deviation amount, the correction amount calculation unit 66 calculates the vertical deviation amount dj based on information from the specific area parallax acquisition unit 61, the ideal parallax acquisition unit 62, and the layout information acquisition unit 64. Equation 5 is used to calculate the vertical deviation amount dj. Here, D in Equation 5 Road is the ideal disparity estimated by the ideal disparity estimation unit 5 based on the road surface parameters, D Lineis the parallax on the specific area (white line) acquired by the specific area parallax acquisition unit 61, and θ is the inclination of the specific area on the image.

[0051]

number

[0052] If the specific region has multiple pixels, the vertical displacement dj is calculated for all pixels or for some pixels, and the most likely estimated result is determined from the multiple estimated results. The determination method is to remove outliers using a statistical method and then calculate the average. Alternatively, any other calculation method may be used.

[0053] <Disparity Image Analysis Unit 7> The parallax image analysis unit 7 analyzes the parallax image P stored in the parallax image storage unit 22. D By analyzing the parallax image P, information used for an automatic driving / driving assistance system is output (for example, detection results of a preceding vehicle, a pedestrian, a road edge, etc.). D is generated based on the left and right images corrected using the vertical deviation dj calculated by the vertical deviation calculation unit 6, the parallax image analysis unit 7 can accurately distinguish between paint or shadows on the road surface and low-level obstacles, and can therefore accurately detect free space where a vehicle can travel and low-level objects that have fallen on the road surface.

[0054] (Processing flow for generating parallax images) Next, the processing flow of the parallax image generating unit 3 will be described with reference to FIG.

[0055] First, in step S11, the geometric correction unit 31 reads the left and right images before geometric correction from the captured image storage unit 21.

[0056] In step S12, the geometric correction unit 31 reads the geometric correction information stored in the geometric correction information storage unit 23 and performs geometric correction on the left and right images acquired in step S11. The process in step S12 results in a left image P L and the right image PR is generated.

[0057] In step S13, the geometric correction unit 31 checks whether the vertical deviation amount dj is stored in the vertical deviation amount storage unit 24. If the vertical deviation amount is stored in the vertical deviation amount storage unit 24, the process proceeds to step S14; if not, the process proceeds to step S15.

[0058] In step S14, the geometric correction unit 31 refers to the vertical displacement stored in the vertical displacement storage unit 24 and determines the correction values ​​for the left and right images. If only the latest vertical displacement dj is stored in the vertical displacement storage unit 24, that value (dj) is set as the correction value. If multiple vertical displacements are stored in the vertical displacement storage unit 24, the vertical displacements within a predetermined range from the current time are referenced, outliers are removed using a statistical method, and the average value is set as the correction value. This method is not limited to this, and any method may be used to set the correction value from multiple vertical displacements. The captured image is corrected based on the determined correction value. The correction method is as shown in Figure 3, and R Corrected right image P R ' is generated.

[0059] In step S15, if step S14 is not performed, the left image P L and the right image P R From the parallax image P D If step S14 is passed, the left image P L and corrected right image P R ' to parallax image P D In either case, the parallax is calculated by searching for a common imaging area using template matching between the left and right images. However, this method is not limited to this, and any other parallax calculation method may be used.

[0060] (Processing flow of the vertical deviation amount calculation unit) Next, the processing flow of the vertical deviation amount calculation unit 6 will be described with reference to FIG.

[0061] First, in step S21, the correction possibility determination unit 65 refers to the road surface parameter reliability, and if it is equal to or less than a predetermined threshold, the process ends without calculating the vertical deviation amount. On the other hand, if it is equal to or greater than the predetermined threshold, step S22 is executed.

[0062] In step S22, the processing area determination unit 63 determines a processing area in which the vertical deviation amount is calculated. At this time, the processing area is determined according to the type label and the distance from the sensor unit 1. Alternatively, the processing area may be determined by setting an arbitrary criterion.

[0063] In step S23, the placement information acquisition unit 64 calculates placement information (for example, the slope θ of the diagonal line) for the region determined as the processing region in step S22.

[0064] In step S24, the correction amount calculation unit 66 calculates the vertical deviation amount dj from the arrangement information (slope θ of the diagonal line), the parallax of the specific region, and the value of the ideal parallax.

[0065] If it is determined in step S25 that the vertical deviation amount dj has been calculated for the entire processing region, step S26 is carried out. If the entire processing region has not been processed, step S24 is repeatedly executed.

[0066] In step S26, the vertical deviation amount is finally determined. If multiple vertical deviation amounts dj are calculated, the vertical deviation amount dj is determined by removing outliers using a statistical method and calculating the average. Alternatively, the vertical deviation amount dj may be determined using any other method.

[0067] (Effects of this embodiment) According to the image processing device of this embodiment described above, by correcting the parallax error caused by the vertical misalignment between the left and right cameras with high precision, it is possible to accurately distinguish between control objects such as objects falling on the road surface and road paint, etc., and to accurately detect control objects even in the case of small steps. [Explanation of symbols]

[0068] 100 Image processing device 1 Sensor unit 11 Left camera 12 Right Camera 2 Storage section 21 Captured image storage unit 22 Parallax image storage unit 23 Geometric correction information storage unit 24 Vertical deviation amount memory section 3. Parallax image generation unit 31 Geometric correction section 32 Parallax calculation unit 4 Specific area extraction part 5 Ideal parallax estimator 51 Road surface parameter estimation unit 52 Specific area parallax estimation unit 6 Vertical deviation calculation section 61 Specific area parallax acquisition unit 62 Ideal parallax acquisition unit 63 Processing area determination unit 64 Location information acquisition unit 65 Correction possibility determination section 66 Correction amount calculation section 7 Parallax image analysis unit

Claims

1. a specific area extraction unit that extracts a specific area on the road surface from left and right images captured by an in-vehicle stereo camera; a road surface parameter estimation unit that estimates a road surface structure expressed by Equation 1 using road surface parameters from road surface portions in the left and right images other than the specific region; a specific area disparity estimation unit that calculates an ideal disparity to be obtained in the specific area from the road surface structure; a parallax image generating unit that generates a parallax image in which an actual parallax, which is a difference in imaging position of an identical object, is stored in each pixel by performing template matching on the left and right images; a vertical displacement calculation unit that calculates a vertical displacement amount of the left and right images using an actual parallax of the specific region, an ideal parallax in the specific region, and arrangement information of each image in the specific region; and The specific area disparity estimation unit calculates an ideal disparity for the specific area using Equation 2 that includes the road surface parameters of Equation 1. Y=aZ+b... (Formula 1) D=B / b×(y-f×a / c)... (Formula 2) Y is the vertical axis (vertical direction) of the Cartesian coordinate system with the installation position of the stereo camera as the origin. Z is the horizontal axis (depth direction) of the Cartesian coordinate system with the installation position of the stereo camera as the origin. a is the slope of the road surface when the center of the camera is the origin, b is the section of the road surface when the center of the camera is the origin, D is the ideal parallax, B is the baseline length of the stereo camera, y is the vertical coordinate of a specific region in the image, f is the focal length of the stereo camera, c is the cell size of the stereo camera.

2. 2. The image processing device according to claim 1, the parallax image generation unit corrects optical distortion of the images captured by the stereo camera; Correcting vertical deviation of the plurality of images captured by the stereo camera based on the vertical deviation amount. An image processing device characterized by:

3. 3. The image processing device according to claim 2, The image processing device is characterized in that the parallax image generating unit determines whether or not to perform correction and the amount of correction based on time-series information on vertical deviation amounts estimated in the past.

4. 3. The image processing device according to claim 2, The specific area extraction unit extracts a diagonal line area on a road surface.

5. 3. The image processing device according to claim 2, The image processing device is characterized in that the specific area extraction unit extracts at least one linear area from among white lines on a road surface, road paint, and shadows.

6. 4. The image processing device according to claim 3, The image processing device is characterized in that the road surface parameter estimation unit estimates road surface parameters from images captured by the stereo camera, and the specific area disparity estimation unit estimates an ideal disparity for the specific area based on the road surface parameters.

7. 6. The image processing device according to claim 5, The image processing device is characterized in that the position information is an inclination angle of the specific area in the image.

8. 8. The image processing device according to claim 7, The image processing device is characterized in that the vertical deviation calculation unit determines whether the vertical deviation amount can be calculated based on the reliability of the road surface parameters.

9. extracting a specific area on a road surface from left and right images captured by an on-board stereo camera; a step of estimating a road surface structure expressed by Equation 1 using road surface parameters from road surface portions in the left and right images other than the specific region; determining an ideal disparity to be obtained in the specific area from the road surface structure; generating a disparity image in which an actual disparity, which is a difference in imaging position of the same object, is stored in each pixel by performing template matching on the left and right images; calculating a vertical shift amount of the left and right images using the actual parallax of the specific region, the ideal parallax in the specific region, and arrangement information of each image in the specific region; and In the step of calculating the ideal parallax, the ideal parallax of the specific area is calculated using Equation 2 including the road surface parameters of Equation 1. Y=aZ+b... (Formula 1) D=B / b×(y-f×a / c)... (Formula 2) Y is the vertical axis (vertical direction) of the Cartesian coordinate system with the installation position of the stereo camera as the origin. Z is the horizontal axis (depth direction) of the Cartesian coordinate system with the installation position of the stereo camera as the origin. a is the slope of the road surface when the center of the camera is the origin, b is the section of the road surface when the center of the camera is the origin, D is the ideal parallax, B is the baseline length of the stereo camera, y is the vertical coordinate of a specific region in the image, f is the focal length of the stereo camera, c is the cell size of the stereo camera.

Images