On-board image processing device

Abstract

Provided is an on-board image processing device capable of performing high-speed processing of three-dimensional measurement while maintaining on-road obstacle detection performance, that is, quickly and accurately detecting the position of a structure or an obstacle on a road. The on-board image processing device comprises: a reduction processing unit 104 that reduces a standard image and a reference image that are captured by a plurality of imaging devices mounted on a vehicle to generate a reduced standard image and a reduced reference image; a first parallax generation unit 105 that generates first parallax information of the reduced standard image and the reduced reference image; a planar region estimation unit 106 that estimates a planar region in at least one of the reduced standard image and the reduced reference image on the basis of the first parallax information; a search range setting unit 107 that sets a search range according to the planar region; and a second parallax generation unit 108 that generates second parallax information by searching the standard image and the reference image within the search range.

Description

In-vehicle image processing device

[0001] The present invention relates to an in-vehicle image processing device for an in-vehicle camera that constitutes an in-vehicle camera system that uses a plurality of imaging devices (cameras) in combination to recognize the environment outside the vehicle.

[0002] JP 2011-121666 A describes the problem as "an object of the present invention is to provide an image processing device and an image processing method capable of quickly and stably acquiring parallax for a desired region of a subject." and describes as a solution the problem as "an image processing device according to a first aspect of the present invention includes: an image input unit that inputs a plurality of images obtained by photographing a single subject from a plurality of viewpoints; a reduced image generation unit that reduces the plurality of images to generate a plurality of reduced images; a first parallax calculation unit that calculates a first parallax by searching for corresponding positions among the plurality of reduced images; a geometric region extraction unit that extracts geometric regions in the plurality of reduced images based on the first parallax and pixel positions of the plurality of reduced images; a display unit that displays the extracted geometric regions in association with the plurality of images; a region designation unit that detects a user's instruction input for the displayed plurality of images and designates a partial region of the plurality of images as a processing region; and a second parallax calculation unit that calculates a second parallax by searching for corresponding positions among the plurality of images for the designated processing region."

[0003] International Publication No. 2017 / 187950

[0004] However, the above-mentioned Patent Document 1 does not take into consideration the high-speed and accurate detection of structures and obstacles on the road.

[0005] Therefore, the present invention aims to provide an on-board image processing device that can quickly and accurately detect the positions of structures and obstacles on the road, even when installed on a four-wheeled vehicle or a two-wheeled vehicle that experiences large posture changes.

[0006] In order to achieve the above-mentioned object, the in-vehicle image processing device of the present invention includes a reduction processing unit that reduces a reference image and a reference image captured by a plurality of imaging devices mounted on a vehicle to generate a reduced reference image and a reduced reference image; a first parallax generation unit that generates first parallax information of the reduced reference image and the reduced reference image; a planar area estimation unit that estimates a planar area in at least one of the reduced reference image or the reduced reference image based on the first parallax information; a search range setting unit that sets a search range according to the planar area; and a second parallax generation unit that generates second parallax information by searching the reference image and the reference image within the search range.

[0007] According to the present invention, the lateral positions of structures and obstacles installed on the road surface can be accurately determined while reducing the amount of calculation.

[0008] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0009] FIG. 1 is a system configuration diagram of a sensing system according to an embodiment of the present invention. FIG. 2 is a detailed diagram of a calculation processing unit of a sensing system according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of distance on an image projection plane obtained from a plane equation. FIG. 4 is an explanatory diagram of stereo matching processing. FIG. 5 is a detailed diagram of a calculation processing unit of a sensing system according to an embodiment of the present invention, which accurately determines the height of a three-dimensional object. FIG. 6 is an explanatory diagram of object positions that can be detected using second and third parallaxes. FIG. 7 is a detailed diagram of a calculation processing unit of a sensing system according to an embodiment of the present invention, which accurately determines the road plane. FIG. 8 is a flowchart of processing of the calculation processing unit of a sensing system according to an embodiment of the present invention.

[0010] An embodiment of the present invention will now be described with reference to the accompanying drawings. In the drawings, parts having the same functions or configurations are designated by the same reference numerals, and repeated description may be omitted.

[0011] FIG. 1 is a diagram showing the system configuration of a sensing system according to an embodiment of the present invention.

[0012] The sensing system 500 of this embodiment is a camera system mounted on a vehicle 505 such as a four-wheeled or two-wheeled vehicle, and is composed of a stereo camera 501, a calculation processing unit 502, a steering unit 503 that controls the direction of movement (steering), and an acceleration / deceleration control unit 504 that controls acceleration / deceleration, and performs collision avoidance control to detect an obstacle 510 on the road and control acceleration / deceleration and the direction of movement (steering) to avoid a collision with the obstacle 510.

[0013] The stereo camera 501 is composed of multiple cameras 101 and 102 mounted on a vehicle 505. The cameras 101 and 102 are sensors that capture images of the surroundings of the vehicle (in this example, the area in front of the vehicle) and output captured image data (images), and are attached to the vehicle 505 in this embodiment so as to monitor the area in front of the vehicle. The cameras 101 and 102 are arranged so that the image capture areas that recognize the environment outside the vehicle have overlapping image capture areas where at least a portion of the image capture areas overlap.

[0014] FIG. 2 shows the details of the calculation processing unit 502.

[0015] The calculation processing unit 502 constitutes the in-vehicle image processing device of the present invention, and is connected to the image acquisition unit 100 that acquires images captured by the cameras 101 and 102, the geometric correction unit 103, the reduction processing unit 104, the first parallax generation unit 105, the planar area estimation unit 106, the search range setting unit 107, and the second parallax generation unit 108 via the bus 1. The calculation processing unit 502 also includes a communication unit (not shown), which is connected to the steering unit 503 and the acceleration / deceleration control unit 504.

[0016] The cameras 101 and 102 are stereo cameras 501, which are calibrated in advance. The geometric correction unit 103 rectifies the left and right images captured by the synchronized cameras 101 and 102. The geometric correction unit 103 outputs an image rectified from the left and right images using calibration parameters prepared in advance.

[0017] Here, we will explain the well-known stereo matching technique. Cameras are placed at multiple positions to capture images of the same target object from multiple different viewpoints, and the distance to the target object is calculated from the difference in appearance between the obtained images, i.e., the parallax. In a typical stereo camera system using two cameras, this conversion is expressed by the following Equation 1. [Equation 1]

[0018] Here, Z [mm] is the distance to the target object, f [mm] is the focal length, w [mm / px] is the pixel pitch, B [mm] is the distance between the cameras (baseline length), and D [px] is the parallax.

[0019] To calculate the parallax, images captured from multiple positions are aligned horizontally, and a search is made to find the position in the right image where the same point is captured for a specific point in the left image. To perform this search efficiently, a parallelization process is generally performed beforehand.

[0020] Rectification refers to vertical alignment of images. In other words, it is a process of calibrating images so that the same points in the left and right images are captured at the same height. In rectified images, only one horizontal row needs to be searched when searching for corresponding points, resulting in high processing efficiency.

[0021] The distance for each pixel can be calculated by searching for matching positions on the left and right sides of each block of the rectified left and right images. Obstacles can be detected by estimating a plane from this distance data, removing the points on the plane, and grouping the remaining points that are close in distance. In addition, for small obstacles such as falling objects, one method is to detect the location of a step from a smooth plane, since the road on the path of travel can be assumed to be a smooth plane.

[0022] Here, a prerequisite for searching for matching positions in the left and right images is that the left and right images must have been captured in the same way. Another prerequisite is that the cut-out block has a sufficiently unique pattern. If the cut-out block does not have a unique pattern, searching the image will find many similar locations, making it impossible to correctly identify the same location in the left and right images. When a car is moving, the image moves smoothly across the screen, so light that should be focused on one pixel of the CMOS sensor may be captured across multiple pixels. This is determined by the exposure time, the distance from the camera to the object, and the relative moving speed of the object and the camera.

[0023] When a sufficiently unique pattern cannot be obtained within a block, the correct location can be identified by increasing the block size and matching against a larger area. However, increasing the block size increases the size of the image to be processed, increasing processing costs. Therefore, by providing a mechanism to reduce the size of a large area in which a sufficiently unique pattern appears within a block to a size that leaves a unique pattern, it is possible to maintain the accuracy of identifying the correct matching position while reducing computational costs.

[0024] The reduction processing unit 104 reduces the images captured by the cameras 101 and 102, respectively. The reduction ratio is a predetermined ratio and is set so that the same ratio is achieved for each of the cameras 101 and 102. The reduction method uses the average pixel method. The average pixel method reduces the image by mixing pixels at a specified aspect ratio. For example, in the case of a 1 / 2 ratio, the average value of four pixels (two pixels vertically and two pixels horizontally) is calculated and output. By calculating the average value in this way, information can be aggregated and reduced. In addition to reduction, there is also a method of reducing the image size by cropping. However, in this case, since we want to obtain approximate distance information for the entire screen, reduction rather than cropping is appropriate. Furthermore, to reduce processing costs, the geometric correction unit 103 and the reduction processing unit 104 may be executed simultaneously. In this case, the geometric correction table can be a table that assumes the reduction ratio, thereby reducing the amount of calculation and memory.

[0025] The first parallax generation unit 105 calculates parallax using the left and right images reduced by the reduction processing unit 104 as input. Here, the right image is used as the base image and the left image as the reference image, and the base image is divided into small regions called blocks. The parallax calculation process involves finding similar small regions of the same size in the reference image. Because the left and right images output by the reduction processing unit 104 are rectified by the geometric correction unit 103, the correlation between the small regions is calculated by shifting the same rows as the base image by one pixel. The correlation is calculated using the Sum of Absolute Difference (SAD). SAD is the sum of the absolute values ​​of the differences between each pixel, and is an index whose value approaches 0 if the images are similar. Due to the nature of parallax, objects located far away have small parallax, so it is sufficient to search the reference image near the same coordinates as the small regions in the base image. On the other hand, objects located close to each other have large parallax, so it is necessary to search the reference image near coordinates distant from the small regions in the base image. In this way, in order to be able to search for the parallax of both distant objects and nearby objects, the parallax search process shifts the reference image horizontally by one pixel at a time based on the coordinates of the base image, and determines the search width so that close-range objects can be captured. Here, because the image is reduced in size by the reduction processing unit 104, the search width set so that the parallax of both distant and nearby objects can be obtained can be reduced. Reducing the search width reduces processing costs, allowing for effective use of the limited amount of calculation.

[0026] The planar area estimation unit 106 performs processing to estimate a planar area from the parallax image obtained by the first parallax generation unit 105. As a method for estimating a plane in three-dimensional space, the coefficients of the plane equation are calculated to minimize the sum of distances between the three-dimensional coordinate points and each pixel obtained from the parallax image obtained by the first parallax generation unit 105 and the plane represented by the plane equation. Considering use in an in-vehicle environment, the road plane often has the largest area, so the plane with the smallest distance from the plane is the road plane (the horizontal surface of the road surface). Here, points with a large distance are excluded as outliers from the coefficients of the calculated plane equation, and the coefficients of the plane equation are calculated again for the remaining points. Repeating this operation several times can reduce the influence of points that are far from the plane.

[0027] Here, we will explain the distance on the image projection plane obtained from the plane equation using an example in which the orientation of the camera mounted on the vehicle is known and it is installed horizontally. If the road is flat, the plane will pass through the origin 0, and the equation will be (a, b, c) = (1, 0, 1). However, in reality, the coefficients change depending on changes in the vehicle's orientation and changes in the gradient of the road surface. If we focus only on gradient changes, for an inclination angle α as shown in the right diagram of Figure 3, this can be expressed by the equation on the right of Figure 3. Once the positional relationship between the camera and the plane can be formulated in this way, it can be projected onto the image coordinate system.

[0028] The search range setting unit 107 calculates the distance for each pixel when a plane is assumed using the coefficients of the plane equation estimated by the plane area estimation unit 106. If the mounting position of the camera attached to the vehicle and the focal length of the lens are known and it is assumed to be a pinhole camera, the distance to the plane for each pixel of the image sensor can be calculated. Specifically, the angle between the plane and the camera is calculated using the coefficients of the plane equation, and the camera coordinate system is converted to screen coordinates.

[0029] This distance is converted into a parallax, and this parallax is used as the reference position of the search range. The search range setting unit 107 sets a position offset by several pixels from the reference position of the search range for each pixel as the search start position in FIG. 4. The reference position set by the search range setting unit 107 contains information on all pixels, making it possible to represent any relationship between the camera and the plane, and is suitable for vehicles with large attitude changes. On the other hand, since it is necessary to store memory for the image, when using a small memory for low cost, one value is stored per row. Because vehicle attitude changes significantly in the pitching direction, storing one value per row makes it difficult to deviate from the reference position of the search range, and therefore, accurate parallax can be calculated. By estimating the planar area using the planar area estimation unit 106, it is possible to handle vehicles with large attitude changes, such as motorcycles.

[0030] The second parallax generation unit 108 receives the reference position output by the search range setting unit 107 and the unscaled left and right images output by the geometric correction unit 103 as inputs, and searches the left and right images within a search range to calculate parallax. One of the unscaled left and right images output by the geometric correction unit 103 is used as a reference image, and the other is used as a reference image. At this time, the combination of the reference image and the reference image used by the first parallax generation unit 105 is used. The search start position is set by offsetting several pixels from the reference image. The search range is also set to a small value of several pixels. This reduces the number of searches and enables accurate calculation of parallax near a plane. Parallax of obstacles that are far from the plane cannot be calculated because they fall outside the search range. When used in an in-vehicle environment, obstacles are often placed on the road surface, and determining whether an obstacle is convex on the road surface is sufficient to determine whether it is an obstacle.

[0031] As a result, it is possible to determine the presence, distance, and position of a certain convex obstacle on the road surface, and by using this information for vehicle acceleration / deceleration and steering control, which are determined based on the obstacle position information, it is possible to fulfill the obstacle avoidance function.By using a small image with a reduced planar area, stereo matching can be performed by focusing on the distance (parallax) of the road surface area, which reduces calculation costs.

[0032] Next, a method for accurately calculating the height of a three-dimensional object will be described. Fig. 5 shows a detailed diagram of a calculation processing unit that accurately calculates the height of a three-dimensional object. In addition to the configuration of Fig. 1, Fig. 5 includes an object detection unit 209, a search width setting unit 210, and a third parallax calculation unit 211.

[0033] The object detection unit 209 uses the UV disparity method for detection. Specifically, it generates a histogram for each horizontal direction (U) of the disparity image and detects the position of the obstacle by peak detection. Using the second disparity as input, this well-known UV disparity method can identify the position of an obstacle (an object on the road) that is convex to the road surface. However, while this method can identify all XYZ positions of the obstacle if all disparities are correctly output, the second disparity does not correctly output the disparity for positions away from the ground in the case of tall obstacles. Because the disparity for positions close to the ground is correctly calculated, the position other than the height is determined by the UV disparity method (Figure 6). Insufficient height information, such as for a cubic-shaped vehicle like the one on the right side of Figure 6, does not affect the collision warning and avoidance control of the vehicle. However, in the case of an obstacle like the tree on the left side of Figure 6, specifically an obstacle that is narrow on the road surface side and wide at positions away from the road surface, the lateral position of the obstacle cannot be correctly detected, and correct avoidance is not possible.

[0034] Therefore, the search width setting unit 210 outputs a search range for re-searching for parallax using a portion of the image output by the geometric correction unit 103, based on the detection frame detected by the object detection unit 209, and a search start position and search width for calculating the parallax. The detection frame detected by the object detection unit 209 is output as the search range for re-searching, and the search width is set to a range including a certain depth width based on the median value of the parallax included in the detection frame. By taking the median value of the parallax included in the detection frame, the parallax value can be extracted without being affected even if some noise is included in the detection frame. For example, assuming that the same obstacle is up to 1 m deep, the search width is set to a range including parallaxes of +1 m and -1 m from the median value of the parallax included in the detection frame. From Equation 1, distance and parallax are inversely proportional, meaning that even if the distance is the same 1 m, the search width becomes larger at closer distances and smaller at longer distances. Therefore, the required search width varies depending on the distance (i.e., the position of the object on the road surface).

[0035] The third disparity calculation unit 211 calculates disparity by searching the left and right images using parameters set by the search width setting unit 210 within a processing area expanded perpendicular to the road surface based on the detection results of the object detection unit 209. Here, the disparity search begins with the area closest to the road surface among the detection results of the object detection unit 209. The expanded processing area is processed, and the processing at the left and right edges of the area will be described in detail below. At the left and right edges, the disparity is also calculated for adjacent areas, and it is confirmed whether the disparity values ​​are the same. The processing continues while expanding the left and right edge areas until the disparity value changes. Furthermore, when calculating disparity in the direction away from the road surface, the processing is terminated at the point where the disparity values ​​for all of a line change significantly. This processing allows accurate calculation of the horizontal disparity, even for an obstacle such as the tree on the left side of Figure 6, which is detected by the object detection unit 209 as a narrow obstacle on the road surface. By executing the processing of the object detection unit 209 again using the updated disparity, accurate obstacle output becomes possible. Furthermore, in order to reduce processing time, the obstacle detection result of the object detection unit 209 may be corrected based on how far the left and right edge regions have been expanded and the processing termination point at which the disparity values ​​for all of one line have changed significantly. By terminating the processing at the point at which the disparity values ​​for all of one line have changed significantly, the region for which disparity is calculated can be minimized, leading to reduced processing costs.

[0036] As described above, by calculating the parallax in three stages, it is possible to reduce the processing load and accurately detect an obstacle on the road surface, including its width.

[0037] Next, a method for accurately determining the road plane will be described. Fig. 7 shows a detailed diagram of a calculation processing unit for accurately determining the road plane. Fig. 7 includes an area discrimination unit 312 in addition to the configuration of Fig. 5.

[0038] When the proportion of obstacles in the image is high and the proportion of the road surface is low, such as when passing each other on a narrow road, road plane estimation is difficult. Therefore, the area identification unit 312 identifies the road surface on which the vehicle 505 is traveling and the flat area outside the road surface. For example, machine learning is used as the identification method.

[0039] The search range setting unit 107 sets the search range based on the disparity that exists within the region of the road surface, among the disparities calculated by the first disparity generation unit 105. Note that instead of region identification, the travel path of the vehicle 505 may be estimated from the steering angle, yaw rate, etc., and the search range may be set using the first disparity limited to the periphery of the travel path estimation result.

[0040] As a result, even when the image contains a large proportion of obstacles and a small proportion of road surface, making it difficult to estimate the road plane, by combining this with free space detection and accurately determining the road plane, it is possible to reduce the processing load and accurately detect obstacles on the road surface, including their width.

[0041] FIG. 8 shows an example of a flowchart of the processing of the calculation processing unit.

[0042] In S800, images (base image and reference image) captured by multiple cameras 101 and 102 mounted on the vehicle (multiple cameras arranged so that the imaging areas that recognize the environment outside the vehicle have overlapping imaging areas at least partially overlapping) are acquired (image acquisition unit 100).

[0043] In S801, the acquired standard image and reference image are rectified (geometric correction unit 103).

[0044] In S802, the rectified standard image and reference image are reduced to generate a reduced standard image and a reduced reference image (reduction processing unit 104).

[0045] In S803, first parallax information of the reduced base image and the reduced reference image is generated (first parallax generating unit 105).

[0046] In S804, a planar area (horizontal surface of the road surface) is estimated in at least one of the reduced base image and the reduced reference image based on the first parallax information (planar area estimation unit 106).

[0047] In S805, the search range is set (limited) according to the flat area (horizontal surface of the road surface) (search range setting unit 107).

[0048] In S806, the standard image and the reference image are searched within the search range (horizontal plane of the road surface) to generate second parallax information (second parallax generating unit 108).

[0049] In S807, an object existing on the road surface is detected based on the second parallax information (object detection unit 209).

[0050] In S808, the search range (search width) is set (limited) in accordance with the road contact position of the object detected in S807 (search width setting unit 210).

[0051] In S809, the standard image and the reference image are searched within the search range determined in S808 to obtain third parallax information (third parallax calculation unit 211).

[0052] According to this embodiment, the lateral positions of structures and obstacles installed on the road surface can be accurately determined while suppressing the amount of calculation.

[0053] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0054] The above components may be configured in part or in whole by hardware, or may be configured to be realized by a processor executing a program. Also, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0055] REFERENCE SIGNS LIST 100 Image acquisition unit 101, 102 Camera 103 Geometric correction unit 104 Reduction processing unit 105 First parallax generation unit 106 Planar area estimation unit 107 Search range setting unit 108 Second parallax generation unit 209 Object detection unit 210 Search width setting unit 211 Third parallax calculation unit 312 Area identification unit 500 Sensing system 501 Stereo camera 502 Calculation processing unit (vehicle-mounted image processing device) 505 Vehicle

Claims

1. An in-vehicle image processing device comprising: a reduction processing unit that reduces a reference image and a reference image captured by a plurality of imaging devices mounted on a vehicle to generate a reduced reference image and a reduced reference image; a first disparity generation unit that generates first disparity information of the reduced reference image and the reduced reference image; a planar area estimation unit that estimates a planar area in at least one of the reduced reference image or the reduced reference image based on the first disparity information; a search range setting unit that sets a search range in accordance with the planar area; and a second disparity generation unit that generates second disparity information by searching the reference image and the reference image within the search range.

2. An in-vehicle image processing device according to claim 1, wherein the plane area estimation unit estimates the horizontal plane of the road surface imaged by the imaging device.

3. An in-vehicle image processing device as described in claim 1, further comprising: an object detection unit that detects an object present on the road surface based on the second disparity information; and a third disparity calculation unit that searches the standard image and the reference image within a search range determined according to the road surface position of the object detected by the object detection unit to obtain third disparity information.

4. An in-vehicle image processing device as described in claim 1, further comprising an area identification unit that identifies, among the road surface imaged by the imaging device, the road surface on which the vehicle is traveling and a flat area outside the road surface, and the search range setting unit sets the search range based on disparity information present within the area of ​​the road surface among the first disparity information.

5. An in-vehicle image processing device according to claim 1, wherein the search range setting unit sets the search range based on the first disparity information limited to the periphery of the estimated path of travel of the vehicle.

Images