On-vehicle image processing apparatus

Abstract

To provide an on-vehicle image processing apparatus which can carry out, at a high speed, three-dimensional measurement keeping its performance on detection of an obstacle on a road, i.e., can detect a position of a structure and / or an obstacle on a road at high speed and with satisfactory accuracy.SOLUTION: An apparatus according to the present invention has a size reduction processing unit 104 for generating a reduced-size standard image and a reduced-size reference image by downsizing a standard image and a reference image obtained by capturing images by a plurality of imaging apparatus mounted on a vehicle, a first parallax generating unit 105 for generating a first parallax information of the reduced-size standard image and the reduced-size reference image, a plane region estimating unit 106 for estimating a plane region in at least either of the reduced-size standard image and the reduced-size reference image based on the first parallax information, a search range setting unit 107 for setting a search range in response to the plane region, and a second parallax information generating unit 108 for generating a second parallax information by searching the search range for the standard image and the reference image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an in-vehicle image processing device for an in-vehicle camera that constitutes an in-vehicle camera system that recognizes the outside world of the vehicle by using a plurality of imaging devices (cameras) in combination. [Background technology]

[0002] Patent Document 1 describes the problem as "an object of the present invention is to provide an image processing device and an image processing method capable of quickly and stably acquiring parallax for a desired region of a subject." and describes the solution as "an image processing device according to a first aspect of the present invention, comprising: an image input unit for inputting a plurality of images obtained by photographing one subject from a plurality of viewpoints; a reduced image generating unit for reducing the plurality of images to generate a plurality of reduced images; a first parallax calculating unit for calculating a first parallax by searching for corresponding positions between the plurality of reduced images; a geometric region extracting unit for extracting a geometric region in the plurality of reduced images based on the first parallax and pixel positions of the plurality of reduced images; a display unit for displaying the extracted geometric region in association with the plurality of images; a region designating unit for detecting a user's instruction input for the displayed plurality of images and designating a portion of the plurality of images as a processing region; and a second parallax calculating unit for searching for corresponding positions between the plurality of images for the designated processing region and calculating a second parallax." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 187950 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned Patent Document 1 does not take into consideration the high-speed and accurate detection of structures and obstacles on the road.

[0005] Therefore, an object of the present invention is to provide an on-board image processing device that can quickly and accurately detect the positions of structures and obstacles on the road, even when installed on a four-wheeled vehicle or a two-wheeled vehicle that has large posture changes. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the in-vehicle image processing device of the present invention includes a reduction processing unit that reduces a reference image and a reference image captured by a plurality of imaging devices mounted on a vehicle to generate a reduced reference image and a reduced reference image, a first disparity generation unit that generates first disparity information of the reduced reference image and the reduced reference image, a planar area estimation unit that estimates a planar area in at least one of the reduced reference image or the reduced reference image based on the first disparity information, a search range setting unit that sets a search range in accordance with the planar area, and a second disparity generation unit that generates second disparity information by searching the reference image and the reference image within the search range. Effect of the Invention

[0007] According to the present invention, the lateral positions of structures and obstacles installed on the road surface can be accurately determined while suppressing the amount of calculation.

[0008] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a system configuration diagram of a sensing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a detailed diagram of a calculation processing unit of the sensing system according to an embodiment of the present invention. [Diagram 3] 1 is an explanatory diagram of the distance on the image projection plane obtained from the plane equation. [Figure 4] FIG. 4 is an explanatory diagram of stereo matching processing. [Diagram 5] FIG. 2 is a detailed diagram of a calculation processing unit of a sensing system for accurately determining the height of a three-dimensional object according to an embodiment of the present invention. [Figure 6] FIG. 13 is an explanatory diagram of an object position that can be detected using the second and third parallaxes. [Figure 7] FIG. 2 is a detailed diagram of a calculation processing unit of a sensing system for accurately determining a road plane according to one embodiment of the present invention. [Figure 8] 5 is a flowchart showing the processing of a calculation processing unit of a sensing system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, parts having the same function or configuration are designated by the same reference numerals, and repeated description may be omitted.

[0011] FIG. 1 is a system configuration diagram of a sensing system according to an embodiment of the present invention.

[0012] The sensing system 500 of this embodiment is a camera system mounted on a vehicle 505 such as a four-wheeled or two-wheeled vehicle, and is composed of a stereo camera 501, a calculation processing unit 502, a steering unit 503 that controls the direction of movement (steering), and an acceleration / deceleration control unit 504 that controls acceleration / deceleration, and performs collision avoidance control to detect obstacles 510 on the road and control acceleration / deceleration and the direction of movement (steering) to avoid collision with the obstacle 510.

[0013] The stereo camera 501 is composed of multiple cameras 101, 102 mounted on a vehicle 505. The cameras 101, 102 are sensors that capture images of the surroundings of the vehicle (in this example, the area in front of the vehicle) and output captured image data (images), and are attached to the vehicle 505 in this embodiment so as to monitor the area in front of the vehicle. The cameras 101, 102 are arranged so that the imaging areas for recognizing the environment outside the vehicle have at least a partial overlapping imaging area.

[0014] FIG. 2 shows the details of the calculation processing unit 502.

[0015] The calculation processing unit 502 constitutes the in-vehicle image processing device of the present invention, and is connected to the image acquisition unit 100 that acquires images captured by the cameras 101 and 102, the geometric correction unit 103, the reduction processing unit 104, the first parallax generation unit 105, the planar area estimation unit 106, the search range setting unit 107, and the second parallax generation unit 108 via the bus 1. The calculation processing unit 502 also includes a communication unit (not shown), and is connected to the steering unit 503 and the acceleration / deceleration control unit 504.

[0016] The cameras 101 and 102 are stereo cameras 501, which are calibrated in advance. The geometric correction unit 103 rectifies the left and right images captured by the left and right synchronized cameras 101 and 102. The geometric correction unit 103 outputs an image rectified from the left and right images using calibration parameters prepared in advance.

[0017] Here, we will explain the well-known stereo matching technology. Cameras are placed in multiple positions to capture the same target object from multiple different viewpoints, and the distance to the target object is calculated from the difference in appearance in the obtained images, i.e., the parallax. In a typical stereo camera system using two cameras, this conversion is expressed by the following formula 1. [Formula 1] TIFF2025041151000002.tif1340

[0018] Here, Z [mm] is the distance to the target object, f [mm] is the focal length, w [mm / px] is the pixel pitch, B [mm] is the distance between the cameras (baseline length), and D [px] is the parallax.

[0019] To calculate the parallax, images captured from multiple positions are aligned horizontally, and a search is made for a position in the right image where the same point is captured for a specific point in the left image. To perform this search efficiently, a parallelization process is generally performed in advance.

[0020] Rectification means vertical alignment of images. In other words, it is a process of calibrating images so that the same points in the left and right images are captured at the same height. In a rectified image, only one horizontal row needs to be searched when searching for corresponding points, which increases processing efficiency.

[0021] The distance for each pixel can be calculated by searching for matching positions for each block of the rectified left and right images. Obstacles can be detected by estimating a plane from this distance data, removing the points on the plane, and grouping the remaining points that are close in distance. In addition, for small obstacles such as falling objects, there is a method of detecting the position of a step from a gentle plane, since the road on the path of travel can be assumed to be a gentle plane.

[0022] Here, the prerequisite for searching for matching positions in the left and right images is that the left and right images must have been captured in the same way. In addition, it is assumed that the cut-out block has a sufficiently unique pattern. If the cut-out block does not have a unique pattern, many similar places will be found even if the image is searched, and the same place cannot be correctly identified from the left and right images. When a car is traveling, the image moves smoothly across the screen, so light that should be focused on one pixel of the CMOS sensor may be captured across multiple pixels. This is determined by the exposure time, the distance from the camera to the object, and the relative moving speed of the object and the camera.

[0023] When a sufficiently unique pattern cannot be obtained within a block, the correct location can be identified by increasing the block size and matching against a larger area. However, increasing the block size increases the size of the image to be processed, and processing costs increase. Therefore, by providing a mechanism to reduce a large area in which a sufficiently unique pattern appears within a block to a size where a unique pattern remains, it is possible to maintain the accuracy of identifying the correct matching position while reducing calculation costs.

[0024] The reduction processing unit 104 reduces the images captured by the cameras 101 and 102. The reduction ratio is a preset ratio and is set so that the cameras 101 and 102 have the same ratio. The reduction method uses the average pixel method. The average pixel method is a method of reducing the image by mixing pixels at a specified ratio in length and width. For example, in the case of 1 / 2 the length and width, the average value of the values ​​of four pixels, two pixels vertically and two pixels horizontally, is calculated and output. By calculating the average value in this way, the information can be aggregated and reduced. In addition to reduction, there is a method of reducing the size of the image by cutting it out, but in this case, since we want to obtain approximate distance information for the entire screen, reduction rather than cutting out is appropriate. In addition, in order to reduce processing costs, the geometric correction unit 103 and the reduction processing unit 104 may be executed simultaneously. In this case, the amount of calculation and memory can be reduced by making the geometric correction table a table assuming the reduction ratio.

[0025] The first parallax generating unit 105 is a process for calculating parallax using the left and right images reduced by the reduction processing unit 104 as input. Here, the right image is used as the standard image, and the left image is used as the reference image, and the standard image is divided into small regions called blocks. The process of finding similar small regions of the same size as these small regions in the reference image is the parallax calculation process. Since the left and right images output by the reduction processing unit 104 are parallelized by the geometric correction unit 103, the same rows as the standard image are shifted by one pixel at a time to find the correlation between the small regions. The correlation is found using SAD (Sum of Absolute Difference). SAD is the absolute sum of the differences between each pixel, and is an index whose value approaches 0 if they are similar. Due to the nature of parallax, since objects located far away have small parallax, it is sufficient to search the reference image near the same coordinates as the small region of the standard image. On the other hand, since objects located close to each other have large parallax, it is necessary to search the reference image near coordinates distant from the small region of the standard image. In this way, in order to search for the parallax of both distant objects and nearby objects, the parallax search process shifts the reference image horizontally by one pixel at a time based on the coordinates of the base image, and determines the search width so that nearby objects can be obtained. Here, since the image is reduced by the reduction processing unit 104, the search width set so that the parallax of both distant and nearby objects can be obtained can be reduced. If the search width can be reduced, the processing cost can be reduced, and the limited amount of calculation can be effectively utilized.

[0026] The plane area estimation unit 106 is a process for estimating a plane area from the parallax image obtained by the first parallax generation unit 105. As a method for estimating a plane in a three-dimensional space, the coefficients of the plane equation that minimize the sum of the distances between the three-dimensional coordinate points of each pixel obtained from the parallax image obtained by the first parallax generation unit 105 and the plane expressed by the plane equation are obtained. Considering use in an in-vehicle environment, the road plane often has the largest area, so the plane that minimizes the distance from the plane becomes the road plane (horizontal surface of the road surface). Here, points that are far away from the coefficients of the obtained plane equation are excluded as outliers, and the coefficients of the plane equation are obtained again for the remaining points. By repeating this operation several times, the influence of points that are far away from the plane can be reduced.

[0027] Here, we will explain the distance on the image projection surface obtained from the plane equation using an example in which the attitude of the camera mounted on the vehicle is known and it is installed horizontally. If the road is flat, the plane passes through the origin 0, and (a, b, c) = (1, 0, 1). However, in reality, the coefficients change depending on the change in the vehicle's attitude and the change in the gradient of the road surface. If we focus only on the change in gradient, in the case of an inclination angle α as shown in the right diagram of Figure 3, it can be expressed by the equation on the right of Figure 3. If the positional relationship between the camera and the plane can be formulated in this way, it can be projected onto the image coordinate system.

[0028] The search range setting unit 107 calculates the distance for each pixel when a plane is assumed using the coefficients of the plane equation estimated by the plane area estimation unit 106. If the mounting position of the camera mounted on the vehicle and the focal length of the lens are known and it is assumed to be a pinhole camera, the distance to the plane for each pixel of the image sensor can be calculated. Specifically, the angle between the plane and the camera using the coefficients of the plane equation is calculated, and the camera coordinate system is converted to screen coordinates.

[0029] This distance is converted into a parallax, and this parallax is set as the reference position of the search range. The search range setting unit 107 sets a position offset by several pixels from the reference position of the search range of each pixel as the search start position in FIG. 4. The reference position set by the search range setting unit 107 has information on all pixels, so it can express any relationship between the camera and the plane, and is suitable for vehicles with large attitude changes. On the other hand, since it is necessary to hold memory for an image, when using a small memory for low cost, one value is held for each row. Since the attitude change of the vehicle varies greatly in the pitching direction, even if one value is held for each row, it is difficult to deviate from the reference position of the search range, and the correct parallax can be calculated. By estimating the planar area by the planar area estimation unit 106, it is possible to handle vehicles such as two-wheeled vehicles whose attitude varies greatly.

[0030] The second parallax generating unit 108 inputs the reference position output by the search range setting unit 107 and the left and right images that are not reduced in size output by the geometric correction unit 103, and calculates parallax by searching the left and right images within the search range. One of the left and right images that are not reduced in size output by the geometric correction unit 103 is set as a reference image, and the other is set as a reference image. At this time, the combination of the reference image and the reference image used in the first parallax generating unit 105 is set as the same. The search start position is set by offsetting several pixels with respect to the reference image. In addition, the search range is also set to a small value of several pixels. This makes it possible to reduce the number of searches, and the parallax near the plane can be correctly obtained. The parallax of an obstacle that is far from the plane cannot be obtained because it falls outside the search range. When used in a vehicle-mounted environment, obstacles are often installed on the road surface, and if it is possible to determine whether or not they are convex on the road surface, it is sufficient to determine whether or not they are obstacles.

[0031] As a result, it is possible to determine the presence, distance, and position of a certain convex obstacle on the road surface, and by using this information for vehicle acceleration / deceleration and steering control, which is determined based on the obstacle position information, it is possible to fulfill the function of obstacle avoidance. By using a small image with a reduced planar area, stereo matching can be performed by focusing on the distance (parallax) of the road surface area, which reduces calculation costs.

[0032] Next, a method for accurately calculating the height of a three-dimensional object will be described. Fig. 5 shows a detailed diagram of a calculation processing unit that accurately calculates the height of a three-dimensional object. Fig. 5 includes an object detection unit 209, a search width setting unit 210, and a third parallax calculation unit 211 in addition to the configuration of Fig. 1.

[0033] The object detection unit 209 uses detection by the UV disparity method. Specifically, a histogram is generated for each horizontal direction (U) of the disparity image, and the position of the obstacle is detected by peak detection. Using the second disparity as an input, the position of an obstacle (an object present on the road) that is convex on the road surface can be identified by the procedure of this well-known UV disparity method. However, this method can identify all XYZ positions of the obstacle when all disparities are correctly output, but the second disparity does not correctly output the disparity at a position away from the ground in the case of a tall obstacle. Since the disparity at a position close to the ground is correctly obtained, the position other than the height is determined by the above UV disparity method (FIG. 6). When the height information is insufficient, for a cubic shape like the vehicle on the right side of FIG. 6, it does not affect the collision warning and avoidance control of the vehicle. However, in the case of an obstacle like the tree on the left side of FIG. 6, specifically, in the case of an obstacle that is narrow on the road surface side and has a width at a position away from the road surface, the lateral position of the obstacle cannot be correctly detected and it cannot be correctly avoided.

[0034] Therefore, the search width setting unit 210 outputs a search range for re-searching for the parallax to obtain the parallax again using a part of the image output by the geometric correction unit 103 based on the detection frame detected by the object detection unit 209, and a search start position and search width for calculating the parallax. The detection frame detected by the object detection unit 209 is output as the search range for re-searching, and the search width is set to a range including a certain depth width based on the median value of the parallax included in the detection frame. By taking the median value of the parallax included in the detection frame, the parallax value can be extracted without being affected even if some noise is included in the detection frame. For example, when assuming that the same obstacle is up to a depth of 1 m, the search width is set to a range including a parallax of +1 m and a parallax of -1 m from the median value of the parallax included in the detection frame. From Equation 1, the distance and the parallax are inversely proportional to each other, and even if the distance is the same 1 m, the search width becomes larger when the distance is close and smaller when the distance is long. Therefore, the required search width varies depending on the distance (i.e., the position of the object on the road surface).

[0035] The third disparity calculation unit 211 calculates disparity by searching the left and right images with the parameters set by the search width setting unit 210 for a processing area obtained by expanding the detection result of the object detection unit 209 in the direction perpendicular to the road surface. Here, the search for disparity starts from the area close to the road surface among the detection results of the object detection unit 209. The expanded processing area is processed, and the processing of the left and right ends of the area will be described in detail. At the left and right ends, the disparity is calculated for adjacent areas and it is confirmed whether the disparity value is the same. The processing is performed while expanding the left and right end areas until the disparity value changes. In addition, in the calculation of disparity in the direction away from the road surface, the processing is terminated at the point where the disparity value of all the lines has changed significantly. With this processing, even if the obstacle is a tree like the one on the left side of FIG. 6, or if the object detection unit 209 detects it as an obstacle with a narrow width on the road surface, the disparity of the horizontal width can be accurately calculated. By executing the processing of the object detection unit 209 again using the updated disparity, it is possible to output an accurate obstacle. In order to reduce processing time, the obstacle detection result of the object detection unit 209 may be corrected based on how far the left and right end areas could be expanded and the processing termination point at which the disparity values ​​for all of a line have changed significantly. By terminating the processing at the point at which the disparity values ​​for all of a line have changed significantly, the area for which disparity is calculated can be minimized, which leads to reduced processing costs.

[0036] As described above, by calculating the disparity in three stages, it is possible to reduce the processing load and accurately detect an obstacle on the road surface, including its width.

[0037] Next, a method for accurately calculating the road plane will be described. A detailed diagram of a calculation processing unit for accurately calculating the road plane is shown in Fig. 7. Fig. 7 includes an area discrimination unit 312 in addition to the configuration of Fig. 5.

[0038] When the ratio of obstacles in the image is high and the ratio of the road surface is low, such as when passing each other on a narrow road, it is difficult to estimate the road plane. Therefore, the area discrimination unit 312 discriminates the road surface on which the vehicle 505 runs and the planar area outside the road surface. For example, a machine learning method is used as the discrimination method.

[0039] The search range setting unit 107 sets the search range based on the disparity present within the area of ​​the traveling road surface among the disparities calculated by the first disparity generating unit 105. Note that instead of the area identification, the traveling path of the vehicle 505 may be estimated from the steering angle and yaw rate, and the search range may be set using the first disparity limited to the periphery of the traveling path estimation result.

[0040] As a result, even when the image contains a large proportion of obstacles and a small proportion of the road surface, making it difficult to estimate the road plane, by combining this with free space detection to accurately determine the road plane, it is possible to reduce the processing load and accurately detect obstacles that have come into contact with the road surface, including their width.

[0041] FIG. 8 shows an example of a flowchart of the processing of the calculation processing unit.

[0042] In S800, images (base image and reference image) captured by multiple cameras 101, 102 mounted on the vehicle (multiple cameras arranged so that the imaging areas for recognizing the environment outside the vehicle have at least a partial overlapping imaging area) are acquired (image acquisition unit 100).

[0043] In S801, the acquired standard image and reference image are rectified (geometric correction unit 103).

[0044] In S802, the rectified base image and reference image are reduced to generate a reduced base image and a reduced reference image (reduction processing unit 104).

[0045] In S803, first disparity information of the reduced base image and the reduced reference image is generated (first disparity generating unit 105).

[0046] In S804, a planar area (horizontal surface of the road surface) is estimated in at least one of the reduced base image and the reduced reference image based on the first disparity information (planar area estimation unit 106).

[0047] In S805, a search range is set (restricted) according to a flat area (horizontal surface of the road surface) (search range setting unit 107).

[0048] In S806, the standard image and the reference image are searched within a search range (horizontal plane of the road surface) to generate second disparity information (second disparity generating unit 108).

[0049] In S807, an object existing on the road surface is detected based on the second disparity information (object detection unit 209).

[0050] In S808, a search range (search width) is set (limited) in accordance with the road contact position of the object detected in S807 (search width setting unit 210).

[0051] In S809, the standard image and the reference image are searched within the search range determined in S808 to obtain third parallax information (third parallax calculation unit 211).

[0052] According to this embodiment, the lateral positions of structures and obstacles installed on the road surface can be accurately determined while suppressing the amount of calculation.

[0053] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to easily explain the present invention, and the present invention is not necessarily limited to the embodiment having all of the described configurations.

[0054] Each of the above configurations may be partially or entirely configured as hardware, or may be configured to be realized by executing a program on a processor. Also, the control lines and information lines are shown as those considered necessary for the explanation, and not all of the control lines and information lines are shown in the product. In reality, it may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0055] 100 Image acquisition unit 101, 102 Camera 103 Geometric correction section 104 Reduction Processing Unit 105 First parallax generator 106 Planar area estimation part 107 Search range setting section 108 Second Parallax Generator 209 Object detection unit 210 Search width setting section 211 Third Parallax Calculation Unit 312 Area Identification Unit 500 Sensing System 501 Stereo Camera 502 Calculation processing unit (vehicle-mounted image processing device) 505 vehicles

Claims

1. a reduction processing unit that reduces the standard image and the reference image captured by a plurality of imaging devices mounted on the vehicle to generate a reduced standard image and a reduced reference image; a first disparity generating unit that generates first disparity information of the reduced base image and the reduced reference image; a planar area estimation unit that estimates a planar area in at least one of the reduced base image and the reduced reference image based on the first disparity information; a search range setting unit that sets a search range according to the planar area; a second disparity generating unit configured to generate second disparity information by searching the base image and the reference image within the search range.

2. The vehicle-mounted image processing device according to claim 1, The plane area estimation unit estimates a horizontal plane of a road surface imaged by the imaging device.

3. The vehicle-mounted image processing device according to claim 1, an object detection unit that detects an object present on a road surface based on the second disparity information; The vehicle-mounted image processing device further includes a third disparity calculation unit that searches the standard image and the reference image within a search range determined according to a road surface position of an object detected by the object detection unit to obtain third disparity information.

4. The vehicle-mounted image processing device according to claim 1, The vehicle further includes an area discrimination unit that discriminates, from among the road surfaces captured by the imaging device, a road surface on which the vehicle is traveling and a flat area outside the road surface, The search range setting unit sets the search range based on disparity information present within a region of the traveling road surface, among the first disparity information.

5. The vehicle-mounted image processing device according to claim 1, The search range setting unit sets the search range based on the first disparity information limited to a periphery of an estimated path of the vehicle.

Images