Image processing apparatus and image processing method

Abstract

To obtain an image processing device and image processing method that can efficiently measure the distance to an object even in the vicinity of the baseline extension line connecting two cameras. [Solution] The image processing device 10 includes a parallelization processing unit 601 that generates multiple parallelized images by parallelizing multiple images captured by multiple imaging units, cameras 1 and 2; a disparity estimation unit 603 that determines the disparity of the multiple parallelized images; a magnification calculation unit 604 that determines the magnification ratio between a first target region of one parallelized image and a second target region corresponding to the first target region in another parallelized image based on the disparity; and a distance determination unit 605 that determines the distance to an object based on the disparity and the magnification ratio.

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus and an image processing method. [Background technology]

[0002] In realizing autonomous driving and advanced safety driving support systems, cameras that recognize the outside world and detect objects necessary for vehicle navigation, such as obstacles and lane information, are becoming increasingly important. In particular, to improve detection performance, cameras are installed on the vehicle itself to acquire surrounding information and realize recognition functions.

[0003] Cameras that recognize such appearances include monocular cameras and stereo cameras that use multiple cameras. Stereo cameras can measure the distance to an object by utilizing the parallax of overlapping areas captured by two cameras placed at a predetermined distance apart.

[0004] Stereo cameras use multiple cameras to measure distance, allowing for an accurate assessment of the risk of collision with an object ahead. The typical method for calculating distance involves using a single SoC (System on the Chip) to determine the correspondence between images captured by multiple cameras, and then calculating the distance from that correspondence.

[0005] When measuring the distance to an object with a stereo camera, the process of deforming the input images so that objects in two images are aligned on the same line is called parallelization. Parallelization requires processing such as image distortion correction, rotation, and projection transformation. The principle of a stereo camera is to find the areas where the same object is captured in the left and right images and estimate a three-dimensional point cloud from the parallax.

[0006] Various methods are known for improving the accuracy of parallax, which is linked to the accuracy of distance estimation. For example, Patent Document 1 discloses a method that acquires object information according to the results of stereo processing and then performs distance measurement again on that region. In recent years, other methods such as deep learning-based methods have also become known. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Patent No. 6723079 [Overview of the project] [Problems that the invention aims to solve]

[0008] Three-dimensional measurement based on the principle of stereo cameras can be performed not only with a group of cameras positioned in front of the vehicle, but also with a group of cameras positioned on the sides of the vehicle to image the surrounding area. However, when measuring three-dimensional point clouds based on parallax, there is a problem in that three-dimensional point clouds cannot be estimated around the baseline extension of the baseline connecting the cameras. The present invention has been made in view of the above points, and its object is to provide an image processing apparatus and an image processing method capable of estimating a three-dimensional point cloud around a baseline extension line. [Means for solving the problem]

[0009] The image processing apparatus of the present invention, which solves the above problems, A parallelization processing unit that generates multiple parallelized images by parallelizing multiple images captured by multiple imaging units, A disparity estimation unit that determines the disparity of multiple parallelized images, A magnification calculation unit that determines the magnification ratio between a first target region of one parallelized image and a second target region corresponding to the first target region in another parallelized image based on the parallax, The system is characterized by comprising a distance determination unit that determines the distance to an object based on the parallax and the magnification ratio. [Effects of the Invention]

[0010] According to the present invention, the distance to an object can be efficiently measured even around the baseline extension line connecting the two cameras. Further features related to the present invention will become apparent from the description in this specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0011] [Figure 1] A diagram for explaining a multi-camera system. [Figure 2] A diagram for explaining a world coordinate system. [Figure 3] A diagram for explaining an image coordinate system. [Figure 4] A diagram for explaining an imaging range around an extension line of a baseline of two cameras. [Figure 5] A diagram for explaining directions of distances in planar projection and spherical projection. [Figure 6] A functional block diagram showing an example of an image processing apparatus according to the present embodiment. [Figure 7] A functional block diagram showing another example of an image processing apparatus according to the present embodiment. [Figure 8] A diagram for explaining a method of calculating distance based on parallax. [Figure 9] A flowchart showing a method of adopting a distance obtained by a method with a small estimation error of distance in a distance determination unit. [Figure 10] A diagram showing an example of hardware constituting the image processing apparatus according to the present embodiment.

Modes for Carrying Out the Invention

[0012] Next, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] FIG. 1 is a diagram showing a vehicle equipped with a plurality of cameras. Vehicle 101 is equipped with cameras 111 to 115. Front camera 111 is positioned in the upper center of the front windshield and captures images of the area 121 in front of the vehicle. Front camera 111 may be a monocular camera or a stereo camera. Left front camera 112 is positioned on the left mirror and captures images of the left front area 122 of vehicle 101. Right front camera 113 is positioned on the right mirror and captures images of the right front area 123 of vehicle 101. Left roof camera 114 is positioned on the upper part of the left center pillar of the vehicle and captures images of the left side of vehicle 101 over a range 124 from front to rear of the vehicle. Right roof camera 115 is positioned on the upper part of the right center pillar of the vehicle and captures images of the right side of vehicle 101 over a range 125 from front to rear of the vehicle.

[0014] These cameras 111-115 acquire information about surrounding objects such as pedestrians, other vehicles, white lines, and the road surface 3, and estimate and acquire physical quantities such as distance, size, and speed. Based on this information about physical quantities, the control quantities for vehicle 101 are determined, and autonomous driving or driver assistance is implemented.

[0015] Figure 2 shows the world coordinate system, and Figure 3 shows the image coordinate system. In the world coordinate system 201, as shown in Figure 2, the X-axis represents the longitudinal direction of the vehicle 101, the Y-axis represents the vehicle width direction, and the Z-axis represents the vertical direction. In the image coordinate system 301, as shown in Figure 3, the u-axis represents the horizontal direction of the image, and the v-axis represents the vertical direction.

[0016] There are several methods for estimating the distance to an object, but a typical example is when multiple cameras are imaging the same object, and the distance is measured using the principle of triangulation.

[0017] Here, any point on the road in world coordinates is given by X=(X,Y,Z,1) in a homogeneous coordinate system. T Let R and T be the external parameter matrices related to the camera's rotation angle and installation position, respectively, and let P=(R|T). Let K be the internal parameter matrix that manages the camera's internal state, such as its focal length and optical center. Let X be the image coordinates in a homogeneous coordinate system, where u=(u,v,1). TIf we denote the scale parameter as s and the lens projection model as L, then (Equation 1) holds for each camera. The subscript indicates the type of camera.

[0018] (Formula 1): TIFF2026098498000002.tif16122

[0019] Disparity measurement using stereo processing is possible by finding corresponding points in these two images where the same object is visible. In calculating disparity, a process called parallelization is applied to transform the images so that identical points are aligned on the same line. This process simplifies the calculation of disparity because it only requires calculation in one direction. The parallelization of these two cameras is achieved by solving the following equation (Equation 2).

[0020] (Formula 2:) TIFF2026098498000003.tif73117

[0021] Here, there is flexibility in how T' is chosen; basically, any direction is acceptable as long as it is a vector orthogonal to the baseline connecting the two cameras. The choice of T' directly affects how the parallelized image is generated. Below, we will refer to this as the parallelized optical axis.

[0022] Depending on how the parallelized optical axis is positioned, an image with an extremely different resolution ratio from the original may be produced, potentially degrading the accuracy of parallax calculations. Furthermore, it may change whether or not the target object can be observed.

[0023] The image coordinates after parallelization are u rect The projection system onto the parallelized image is L rect Therefore, the coordinates of each image can be expressed by the following relationship.

[0024] (Formula 3): TIFF2026098498000004.tif24117

[0025] By performing parallelization processing on images in this way, efficient searching becomes possible. In addition to the general perspective projection method, there are also projection methods that project onto a disk or spherical projection. Unlike the perspective projection method, the spherical projection method makes it possible to create a parallelized image that includes objects placed on the baseline extension.

[0026] On the other hand, these projection methods have the problem that the distance in the baseline direction cannot be estimated even when using parallax calculation. The principle is as follows. In disk projection and spherical projection, L rect Each is L rect1 , L rect2 It is defined by the following equation (3). Below, x, y, and z are the camera coordinate system, where z is the depth and x and y represent the horizontal and vertical directions, respectively.

[0027] (Formula 4): TIFF2026098498000005.tif51120

[0028] Figure 4 shows an example of the area around the baseline extension. In Figure 4, Camera 1 and Camera 2 schematically represent the right front camera 113 and the right roof camera 115 in Figure 1, respectively. The area on the road surface 3 centered on the intersection 422 where the baseline extension line 421a of the baselines 421 of Camera 1 and Camera 2 intersects with the road surface 3 is defined as the baseline extension area 423. Note that the intersection 422 is not included in the concept of the baseline extension area 423. In Figure 4, the baseline extension area 423 is represented as a circular shape on the road surface 3 centered on the intersection 422 where the baseline extension line 421a of the baselines 421 of Camera 1 and Camera 2 intersects with the road surface 3, but the shape of the baseline extension area 423 is not limited to this.

[0029] Figure 5 illustrates the direction of distance in planar projection and spherical projection. Figure 5(a) shows the perspective projection method, and Figure 5(b) shows the spherical projection method (cylindrical projection method), showing planes 512 and 513 that are the same distance from the baseline 421 in a direction perpendicular to it. Points 522 and 523 are located on the baseline extension line 421a of camera 1 and camera 2. The dashed line direction 511 extends in a direction perpendicular to the baseline 421.

[0030] For images projected using each projection method, the planes at the same distance are, in the case of the disk projection method, a planar distance plane 512 perpendicular to the dashed line direction 511, as shown in Figure 5(a), and in the case of the spherical projection method, a cylindrical distance plane 513 in the direction of the dashed line shown in Figure 5(b). However, in all of these projection methods, the distance in the direction perpendicular to the baseline extension line 421a is 0, and points 522 and 523 located on the baseline extension line 421a overlap and are projected onto the same pixel. Therefore, it is not possible to calculate three-dimensional points on the baseline extension line 421a.

[0031] Thus, while the baseline extension line 421a is not included in the calculation of three-dimensional points, the parallax is small even in the area surrounding the baseline extension line 423, and the accuracy of three-dimensional point estimation deteriorates. Here, the area surrounding the baseline extension line 423 specifically refers to the area around intersections 422 where the vector representing the baseline extension line 421a intersects with objects such as roads or three-dimensional structures.

[0032] Figure 6 is a functional block diagram of the image processing apparatus of this embodiment, and is a configuration diagram for solving the present problem. The image processing device 10 has, internally, a pair of parallelization processing units 601 and 602, a disparity estimation unit 603, a magnification calculation unit 604, and a distance determination unit 605. The pair of parallelization processing units 601 each perform parallelization processing on images captured by a pair of cameras 1 and 2, and generate a pair of parallelized images. The disparity estimation unit 603 estimates the disparity of the pair of parallelized images.

[0033] The magnification calculation unit 604 determines the magnification ratio between a first target region in one parallelized image and a second target region in the other parallelized image that corresponds to the first target region, based on a pair of parallelized images and parallax. The distance determination unit 605 determines the distance to the object using at least one of the first distance to the object determined by parallax or the second distance determined by the magnification ratio.

[0034] For example, as the distance from cameras 1 and 2 to the object increases, the size of the object captured by camera 1 and the size of the object captured by camera 2 become approximately the same. In other words, the magnification ratio approaches 1, and the difference between the distance from camera 1 to the object and the distance from camera 2 to the object becomes smaller. Therefore, the distance determination unit 605 may determine the first distance as the distance to the object when the distance from cameras 1 and 2 to the object is greater than a threshold, and determine the second distance as the distance to the object when the distance from cameras 1 and 2 to the object is closer than a threshold. This improves the accuracy of distance calculation by appropriately using the first and second distances according to the distance to the object. Alternatively, the distance determination unit 605 may use the first and second distances to calculate the distance to the object such that the weight of the second distance increases compared to the first distance as the distance from cameras 1 and 2 to the object increases. When the distance determination unit 605 compares the distance to the object with the threshold, it may use one or both of the first distance and the second distance, or it may use the distance to the object obtained or estimated by other means.

[0035] Furthermore, in the vicinity 423 of the baseline extension, the parallax increases as the distance from the intersection 422 of the baseline extension 421a increases. Therefore, the distance determination unit 605 may calculate the distance to the object in the vicinity 423 of the baseline extension, such that the weight of the first distance increases compared to the second distance as the distance from the intersection 422 of the baseline extension 421a to the object increases. That is, the distance determination unit 605 may weight the first distance and the second distance respectively according to the position of the object on the image relative to the baseline extension 421a, which is a virtual extension of the baselines of cameras 1 and 2, and calculate the distance to the object based on the weighted first distance and the second distance. This improves the accuracy of distance calculation by appropriately using the first and second distances, taking advantage of the fact that the parallax increases as the object moves away from the baseline extension line 421a in the image, that is, the accuracy of the parallax estimated by the parallax estimation unit 603 improves. The distance determination unit 605 may also correct the first distance with the second distance. The distance determination unit 605 may weight the first and second distances respectively based on the parallax error and the magnification ratio error, and determine the distance to the object based on the weighted first and second distances.

[0036] Images input from cameras 1 and 2 are parallelized by parallelization processing units 601 and 602, respectively, and one parallelized image and the other parallelized image are output (first step). The disparity estimation unit 603 takes each parallelized image as input and outputs the difference in position between each parallelized image as a disparity image (second step). The magnification calculation unit 604 takes each parallelized image and the disparity image as input and calculates the magnification ratio based on either the left or right image (third step). Based on the disparity between one parallelized image and the other parallelized image, the magnification calculation unit 604 determines the magnification ratio between the first target area of ​​one parallelized image and the second target area in the other parallelized image that corresponds to the first target area.

[0037] The magnification ratio refers to the magnification that indicates how large a texture or object located at a certain position in the reference image (the first target area) appears in the other image. If it appears at 2x magnification, the magnification ratio is 2.0, and if it appears at 0.5x magnification, it is 0.5. If the reference image is the right image (one parallelized image), the corresponding point in the left image (the other parallelized image) can be determined from the position in the right image and the disparity image, and the magnification ratio is calculated between these corresponding points.

[0038] The magnification factor can be calculated using methods such as the LucasKanade method or deep learning-based methods, but is not limited to these techniques. However, the calculation of this magnification factor only needs to be done for the horizontal (u-axis) magnification of the image. This is because it is performed on parallelized images where the lines of the captured objects are aligned vertically (v-axis) on both the left and right sides, so the vertical magnification will be the same on both sides.

[0039] Since the magnification is calculated only in the horizontal direction of the image, it can be calculated more efficiently than calculating the magnification in both the horizontal and vertical directions. After calculating the magnification, the distance determination unit 605 determines a three-dimensional point from the parallax (d) and magnification (s) in the reference image. This allows the distance to the object to be calculated (step 4).

[0040] Using spherical projection as an example, we show the formulas for calculating a three-dimensional point ((Equations 5), (6), and (7)). However, R is distance, φR and θ is the angle obtained from the pixel which is the measurement position on the spherical projection, and is shown in FIG. 8 together with the axial directions of (x, y, z). FIG. 8 is a diagram for explaining the method of calculating the parallax-based distance. In FIG. 8, reference symbol B indicates the baseline between cameras 1 and 2, reference symbol R indicates the distance from the baseline to the object. And reference symbol θ is the elevation angle from cameras 1 and 2 to the object, and φ R is the inclination angle of the baseline direction to the object with respect to the broken line passing through camera 1 and orthogonal to the baseline B, and φ L is the inclination angle of the baseline direction to the object with respect to the broken line passing through camera 2 and orthogonal to the baseline.

[0041] d in (Equation 6) indicates parallax, R indicates the distance in the broken line direction 511 shown in FIG. 5(b), and similar to the above-described explanation of parallax, the same distances are taken concentrically on the plane. B in (Equation 6) indicates the baseline length. Equation (7) is an equation for calculating the baseline direction distance (x) by the magnification (s), and s in (Equation 7) indicates the magnification.

[0042] (Equation 5) indicates a three-dimensional point, and by using (Equation 6) and (Equation 5), a three-dimensional point according to the parallax can be calculated. By using (Equation 7) and (Equation 5), a three-dimensional point group can be obtained by solving the equation of s for x.

[0043] (Equation 5): TIFF2026098498000006.tif7129(Equation 6): TIFF2026098498000007.tif10129(Equation 7): TIFF2026098498000008.tif32129

[0044] In this way, the calculation of the three-dimensional point can also be carried out in the vicinity 423 of the baseline extension line.

[0045] FIG. 7 shows an example having a function of estimating the measurement errors of parallax and magnification from the calculation results. In this example, the same components as those in FIG. 6 are denoted by the same reference symbols, and the detailed description thereof is omitted. The parallax error estimation unit 606 and the magnification error estimation unit 607 are derived from experimental statistical information, learning-based reliability, and theoretical values, and the method is not limited. Two examples are given below.

[0046] As a first example, we show a case where the parallax error can be statistically estimated in advance. The parallax is mean 0 pixels, and the standard deviation is ε. d Assume the pixels have errors that follow a normal distribution. Furthermore, the magnification is given by a mean of 0 and a standard deviation of ε. s Assume that the error follows a normal distribution of %. These errors are set by the disparity error estimation unit 606 and the magnification error estimation unit 607, respectively, in Figure 7.

[0047] The distance error at each measurement point can be estimated by partially differentiating the respective calculation formulas with respect to d and s, and then using a linear approximation with the standard deviation, so the distance estimation error for each method can be calculated. In the distance determination unit 605 in Figure 7, these calculation results are used to determine the distance obtained using the method with the smallest distance estimation error, or the calculated distances are weighted using the reciprocal of the distance estimation error, and then output.

[0048] Figure 9 is a flowchart showing how the distance determined by the distance determination unit is adopted based on a method that minimizes distance estimation error. The distance determination unit 605 receives the parallax 906, parallax error 909, magnification 901, and magnification error 904, and calculates the distance (three-dimensional point) based on the parallax 906 and the distance (three-dimensional point) based on the magnification 901, respectively, 902 and 907, and similarly performs error estimation 903 and 908. Then, based on the comparison 910 of these estimated distance errors, it determines the final distance (three-dimensional point) 911.

[0049] Similar to the results of distance calculation 902 using magnification (a-1) and distance calculation 907 using parallax (b-1), distance error estimation 903 is performed by the magnification error estimation unit 607 and distance error estimation 908 is performed by the parallax error estimation unit 606. Then, the distance determination unit 605 compares the respective estimation results (a-2)(b-2) with their respective errors 910, performs distance determination 911 by adopting the distance with the smaller error, and outputs it (c).

[0050] As the next example, we will show a case where, when outputting parallax and magnification using methods such as deep learning, the amount of error can be estimated simultaneously during inference. Furthermore, we will assume that invalid values ​​indicating estimation failure can also be output at the same time. In this case, if both estimated values ​​are valid, the respective errors at the measurement point can be estimated by partially differentiating the respective calculation formulas with respect to d and s, and then using a first-order approximation with the standard deviation, similar to the first example, so that the distance estimation errors for each method can be calculated. The distance determination unit 605 determines whether the first distance and the second distance are invalid values ​​based on the parallax error and the magnification error, and determines the distance to the object based on the result of the invalid value determination.

[0051] In the distance determination unit 605 shown in Figure 7, the distance calculated using the method with the smallest distance estimation error is adopted from these calculation results, or the calculated distances are weighted by the reciprocal of the distance estimation error, and the result is output. If one of the calculations is invalid, the valid one is adopted, and if both are invalid, the distance determination unit 605 outputs an invalid value.

[0052] From the measurement error obtained in this way, the distance determination unit 605 outputs a calculation result based on one of the measurement results, invalid values, and an estimated error that takes into account the errors of both error estimation units 606 and 607.

[0053] Figure 10 shows an example of the hardware that constitutes the image processing apparatus of this embodiment. The image processing device 10 has an ECU (Electronic Control Unit) and is mainly composed of a microcomputer consisting of a CPU 11, ROM 12, RAM 13, storage device 14, and input / output interface 15. The CPU 11 is a computer that executes various programs installed in the storage device 14. The ROM 12 and RAM 13 function as main memory units that store various programs, data, etc. necessary for the CPU 11 to execute the various programs stored in the storage device 14. The storage device 14 stores various programs and data. The functions of the parallelization processing units 601, 602, disparity estimation unit 603, magnification calculation unit 604, distance determination unit 605, disparity error estimation unit 606, and magnification error estimation unit 607 of the image processing device 10 are realized by the CPU 11 loading the programs stored in the storage device 14 into the ROM 12 and RAM 13 and executing them.

[0054] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and various design modifications can be made without departing from the spirit of the invention as described in the claims. For example, the embodiments described above are described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add a configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations. [Explanation of Symbols]

[0055] 1, 2...Camera, 10...Image processing device, 601, 602...Parallelization processing unit, 603...Parallax estimation unit, 604...Magnification rate calculation unit, 605...Distance determination unit, 606...Parallax error estimation unit, 607...Magnification rate error estimation unit

Claims

1. A parallelization processing unit that generates multiple parallelized images by parallelizing multiple images captured by multiple imaging units, A disparity estimation unit that determines the disparity of multiple parallelized images, A magnification calculation unit that determines the magnification ratio between a first target region of one parallelized image and a second target region corresponding to the first target region in another parallelized image based on the parallax, An image processing apparatus comprising: a distance determination unit that determines the distance to an object based on the parallax and the magnification ratio.

2. An image processing apparatus according to claim 1, The distance determination unit, The first distance to the object, determined by the parallax, and the second distance, determined by the magnification ratio, are determined as follows: The first distance and the second distance are weighted according to the position of the object in the image, based on the baseline extension line obtained by virtually extending the baselines of the plurality of imaging units. An image processing device that determines the distance to the object based on the weighted first distance and the second distance.

3. An image processing apparatus according to claim 1, The distance determination unit determines either a first distance to the object determined by the parallax or a second distance determined by the magnification ratio as the distance to the object, in an image processing apparatus.

4. An image processing apparatus according to claim 2, A parallax error estimation unit estimates the parallax error obtained by the parallax estimation unit, The system further comprises a magnification error estimation unit that estimates the error in the magnification ratio obtained by the magnification calculation unit, The distance determination unit determines the distance to the object based on the parallax error and the magnification ratio error, and is an image processing apparatus.

5. An image processing apparatus according to claim 4, The distance determination unit determines whether or not to correct the first distance by the second distance based on the parallax error and the magnification ratio error, and is an image processing apparatus.

6. An image processing apparatus according to claim 4, The distance determination unit determines either the first distance or the second distance as the distance to the object based on the parallax error and the magnification ratio error, in an image processing apparatus.

7. An image processing apparatus according to claim 4, Image processing apparatus wherein the distance determination unit weights the first distance and the second distance, respectively, based on the parallax error and the magnification ratio error, and determines the distance to the object based on the weighted first distance and the second distance.

8. An image processing apparatus according to claim 4, Image processing apparatus, wherein the distance determination unit determines whether the first distance and the second distance are invalid values ​​based on the parallax error and the magnification ratio error, and determines the distance to the object based on the result of the invalid value determination.

9. An image processing method for determining the distance to an object by processing multiple images captured by multiple imaging units, The first step is to generate multiple parallelized images by parallelizing the aforementioned multiple images, A second step involves determining the disparity between the plurality of parallelized images generated in the first step, A third step is to determine the magnification ratio between a first target region of one parallelized image and a second target region corresponding to the first target region in the other parallelized image, based on the parallax obtained in the second step. A fourth step in which the distance from the plurality of imaging units to the object is determined based on the parallax obtained in the second step and the magnification ratio obtained in the third step, An image processing method characterized by including

Images