In-vehicle image processing device

Abstract

Provided is an in-vehicle image processing device capable of efficiently measuring the distance to an object. The present invention includes: an image acquisition unit B03 for acquiring images from a plurality of imaging devices for capturing a plurality of images having different viewpoints; a parallelizing optical axis priority determination unit B07 for determining priority for a plurality of optical axis candidates which are arbitrary optical axes on a plane orthogonal to the base line direction of the plurality of imaging devices; a parallelization processing unit B04 for parallelizing and outputting the plurality of images by using at least one optical axis among the optical axis candidates for which the priority is determined; a parallax calculation unit B05 for determining parallax on the basis of the plurality of images outputted from the parallelization processing unit B04; and an object detection unit B06 for detecting an object (in the images) on the basis of the parallax.

Description

In-vehicle image processing device

[0001] The present invention relates to an in-vehicle image processing device suitable for a system that highly understands the surrounding environment in automated driving and advanced safe driving support systems.

[0002] In order to realize autonomous driving and advanced safety driving support systems, cameras that monitor the outside world and detect objects necessary for vehicle navigation, such as obstacles and lane information, are becoming increasingly important.

[0003] In particular, to improve detection performance, the vehicle is equipped with multiple cameras to acquire information about the vehicle's surroundings and realize an external environment recognition function.

[0004] There are two types of cameras that recognize the outside world: a monocular camera and a stereo camera that uses multiple cameras. A stereo camera can measure the distance to an object by using the parallax of the overlapping area between two images taken from different viewpoints by two cameras spaced a predetermined distance apart.

[0005] Stereo cameras use multiple cameras to measure distance, making it possible to accurately grasp the risk of collision with an object ahead. Distance is generally calculated by connecting multiple cameras to a single computing resource such as a System on a Chip (SoC), aggregating images, and then deriving the correspondence between images while accessing them to determine the distance.

[0006] When measuring the distance to an object using the stereo camera principle, the process of transforming input images so that objects in two images are aligned on the same line is called parallelization. Parallelization requires image distortion correction, image rotation, projective transformation, and other processes. In this parallelization process, the optical axis after parallelization must be determined, and this can be any axis that is perpendicular to the baseline connecting the positions of the two cameras, as long as it includes an overlapping area. For example, Patent Document 1 discloses a technique for setting the optical axis of a forward-facing stereo camera to face the road surface after parallelization.

[0007] JP 2019-16308 A

[0008] Three-dimensional measurements based on the stereo camera principle can be performed not only with cameras placed in front, but also with stereo cameras placed in the periphery. Stereo cameras placed in the periphery have a high degree of freedom, as images with large distortions may be combined or the baseline may be slanted, and depending on the optical axis setting, this can affect detection ability and measurement accuracy, resulting in a decrease in measurement performance.

[0009] The present invention has been made in view of the above-mentioned problems, and has an object to provide an in-vehicle image processing device that can efficiently measure the distance to an object.

[0010] In order to solve the above problems, the in-vehicle image processing device of the present invention includes an image acquisition unit that acquires images from a plurality of imaging devices that are mounted on a vehicle and capture a plurality of images from different viewpoints; a parallelization optical axis priority determination unit that determines the priority of a plurality of optical axis candidate locations that are any optical axes on a plane perpendicular to the baseline direction of the plurality of imaging devices; a parallelization processing unit that parallelizes the plurality of images using at least one optical axis of the optical axis candidate locations whose priority has been determined, and outputs the plurality of images; a parallax calculation unit that calculates parallax based on the plurality of images output from the parallelization processing unit; and an object detection unit that detects an object based on the parallax.

[0011] According to the present invention, the distance to an object can be measured efficiently.

[0012] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0013] FIG. 1 is a diagram illustrating a multi-camera system and a coordinate system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating the configuration of an in-vehicle image processing device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating the configuration of a detection range calculation unit. FIG. 4 is a diagram illustrating an example of utilization of the detection range calculation unit. FIG. 5 is a diagram illustrating the configuration of a spatial resolution calculation unit. FIG. 6 is a diagram illustrating the configuration of a measurement accuracy estimation unit. FIG. 7 is a diagram illustrating the configuration of an in-vehicle image processing device according to an embodiment of the present invention, which has a vehicle operation state and a driving assistance function state. FIG. 8 is a diagram illustrating an example of utilization of the detection range calculation unit according to an operation mode. FIG. 9 is a diagram illustrating the configuration of an in-vehicle image processing device according to an embodiment of the present invention, which combines multiple imaging devices.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each drawing for explaining the embodiments of the present invention, parts having the same functions are given the same reference numerals, and repeated description thereof will be omitted.

[0015] FIG. 1 shows the configuration of a multi-camera system A006 equipped with a front camera A001, a left front camera A002, a right front camera A003, a left side camera A004, and a right side camera A005 on the front, left front, right front, left side, and right side of a vehicle A000, respectively. The multi-camera system A006, comprised of these cameras, acquires information about surrounding objects such as pedestrians, vehicles, white lines, and road surfaces, and estimates and acquires physical quantities such as their distance, size, and speed. It then determines vehicle control parameters based on the information to perform automated driving, driving assistance, and the like. The vehicle A000 is equipped with an in-vehicle image processing device B00 ( FIG. 2 ) for implementing the above-described processing. In this embodiment, a vehicle coordinate system (world coordinate system) is used, as shown in A007, with the direction of travel of the vehicle A000 defined as the X-axis, the horizontal direction as the Y-axis, and the vertical direction as the Z-axis. An image coordinate system is used, as shown in A008.

[0016] There are several methods for estimating the distance to an object, but a typical example is when multiple cameras capture images of the same object and measure it using the principle of triangulation.

[0017] Here, any point in the world coordinates on the road is expressed as X = (X, Y, Z, 1) in the homogeneous coordinate system. T The external parameter matrices related to the camera's rotation angle and installation position are R and T, respectively. The matrix is ​​P = (R | T). The internal parameter matrix that manages the internal state of the camera, such as the focal length and optical center, is K. The image coordinates where X is captured are expressed as u = (u, v, 1) in the homogeneous coordinate system. T If the scale parameter is s and the lens projection model is L, the following equation 1 holds for each camera. Note that the subscripts indicate the type of camera. [Equation 1]

[0018] Disparity measurement using stereo processing is possible by finding corresponding points in these two images, but by performing a process called rectification, it is possible to convert the images so that identical points are aligned on the same line on the image. This rectification of the two cameras is achieved by solving the following equation 2: [Equation 2]

[0019] Here, there is a degree of freedom in how T' is chosen, and basically any direction is acceptable as long as it is a vector perpendicular to the baseline connecting the two cameras. The way T' is chosen is directly related to how the image after collimation is generated. Below, this is called the collimation optical axis.

[0020] Depending on how the collimated optical axis is positioned, an image with a significantly different resolution ratio from the original may be created, which may result in a deterioration in the accuracy of parallax calculations, etc. It may also affect whether or not the object to be detected can be observed.

[0021] The image coordinates after rectification are expressed as u rect Let L be the projection system onto the rectified image. rect Then, each image coordinate can be expressed by the following relational expression 3. [Equation 3]

[0022] 2 shows an example of the configuration of an in-vehicle image processing device according to an embodiment of the present invention. The in-vehicle image processing device B00 of this embodiment is composed of an image acquisition unit B03, a parallelization processing unit B04, a parallax calculation unit B05, an object detection unit B06, and a parallelization optical axis priority determination unit B07, and the parallelization optical axis priority determination unit B07 is characterized in that it determines the priority of the parallelization optical axis based on a detection range calculation unit B08, a spatial resolution calculation unit B09, a measurement accuracy estimation unit B10, and a camera parameter group B11 including the camera parameters shown in Equations 1 to 3, and the image size and processing range.

[0023] The image acquisition unit B03 acquires images from a plurality of imaging devices B01, B02 mounted on the vehicle, each imaging device capturing a plurality of images from different viewpoints, and transmits the acquired images to a parallelization processing unit B04. As an example, the plurality of imaging devices B01, B02 are configured by a pair of cameras (imaging devices) that share a part of their field of view or have a part of their field of view overlapping (having an overlapping area) from among the front camera A001, left front camera A002, right front camera A003, left side camera A004, and right side camera A005 shown in FIG. 1 .

[0024] The parallelization processing unit B04 parallelizes the images obtained from the two imaging devices B01 and B02 based on the image information transmitted from the image acquisition unit B03, the camera parameter group B11 updated in advance or during driving, and the optical axis selected by the parallelization optical axis priority determination unit B07, and transmits the parallelization results to the downstream parallax calculation unit B05.

[0025] The disparity calculation unit B05 generates a disparity image based on the multiple images rectified by the rectification processing unit B04 using techniques such as block matching, feature point matching, and deep learning, and transmits it to the downstream object detection unit B06.

[0026] The object detection unit B06 analyzes the parallax image generated by the parallax calculation unit B05 to detect an object.

[0027] The collimation optical axis priority determination unit B07 sets (determines) priorities for the collimation optical axes (plurality of optical axis candidate locations that are any optical axes on a plane perpendicular to the baseline direction of the plurality of imaging devices B01, B02) based on the results of the detection range calculation unit B08, the spatial resolution calculation unit B09, and the measurement accuracy estimation unit B10, and selects, for example, the collimation optical axis with the highest priority and transmits it to the collimation processing unit B04.

[0028] FIG. 3 shows an example configuration of the detection range calculation unit B08. The detection range calculation unit B08 includes a detection object range determination unit C01 that outputs the position of an expected object, and a detection range estimation unit C03 that determines (estimates) whether the position of the object output by the detection object range determination unit C01 falls within the angle of view of the parallax image (object detection range) using information C02, particularly the size and range of the parallax image, from the camera parameter group B11. The collimation optical axis priority determination unit B07 sets (determines) and selects a collimation optical axis that is suitable for satisfying the criteria as having a high priority. That is, the detection range calculation unit B08 determines the object detection range occupied by the object to be detected in the image after collimation processing using multiple optical axis candidates, and the collimation optical axis priority determination unit B07 determines the priority of the collimation optical axis based on the object detection range. The collimation optical axis may be set from among pre-set collimation optical axes using the above criteria, or an optical axis that satisfies the criteria may be analytically selected. The detection object and its range used by the detection object range determination unit C01 are determined based on the position and range of the object detected by the object detection unit B06 and the position and range of the object estimated in time series.

[0029] For example, FIG. 4 shows an example of a scene in which two vehicles are traveling parallel to the host vehicle. In the scene D10, when the collimated optical axis is set as D11, the estimated detection range is set as D12, and only the upper part of the vehicle D13 traveling parallel to the host vehicle D14 is detected. On the other hand, in the scene D20, when the collimated optical axis is set as D21, the estimated detection range is obtained as D22, and the lower part of the vehicle D23 traveling parallel to the host vehicle D24 is detected as the center. Depending on the detection situation, for example, if the optical axis is initially directed in a case such as D10, the optical axis may be changed to D20 if the lower part of the vehicle needs to be accurately measured. This case is effective, for example, when it is necessary to know the exact distance to a vehicle approaching from the side. Specifically, in a case where the object detection unit B06 detects a parallel traveling vehicle D23, the detection target range determination unit C01 estimates an overall image from partial visibility of the vehicle, and the detection range estimation unit C03 determines that the lower part of the vehicle is not visible and sets (determines) and selects the collimated optical axis that can capture the lower part of the vehicle (having a wide object detection range) as having a high priority. The collimation processing unit B04, parallax calculation unit B05, and object detection unit B06 perform collimation processing and measure the distance according to the newly set optical axis.

[0030] 5 shows an example of the configuration of the spatial resolution calculation unit B09. The spatial resolution calculation unit B09 includes a pre-collimation spatial resolution calculation unit E01 that indicates the spatial resolution before collimation, a post-collimation spatial resolution calculation unit E02 that indicates the spatial resolution after collimation, and a spatial resolution change amount calculation unit E03 that calculates the difference in spatial resolution before and after collimation using information from the camera parameter group B11. The collimation optical axis priority determination unit B07 sets (determines) the collimation optical axis that is most suitable according to the result of the spatial resolution change amount calculation unit E03 as having a high priority. Alternatively, the collimation optical axis priority determination unit B07 may set the priority of the collimation optical axis from among the collimation optical axes set in advance using the above information. That is, the spatial resolution calculation unit B09 calculates the change in spatial resolution between the image before collimation and the image after collimation using multiple optical axis candidates, and the collimation optical axis priority determination unit B07 determines the priority of the collimation optical axis based on the change in spatial resolution.

[0031] The pre-collimation resolution is estimated from internal parameters such as the lens distortion model, focal length, and image size. Here, the pre-collimation resolution may simply be returned in units of one pixel without specifying the angle of incidence, for example. Alternatively, the magnitude of the angle of incidence per pixel may be considered. The external parameters of the camera may be used to assume the surface of the object to be detected, and the size of the image plane captured per pixel may also be calculated.

[0032] The post-collimation resolution is estimated from internal parameters after collimation, such as the collimated optical axis, the projection model at the time of collimation, the focal length, and the image size. As with the pre-collimation resolution, the incident angle per pixel can be considered. Alternatively, the pixel quantity per incident angle, which is the reciprocal of the incident angle, can be used as the basis. When using the external parameters of the camera, the size of the imaging surface captured per pixel can be calculated by assuming the surface of the object to be detected.

[0033] The spatial resolution change amount calculation unit E03 is characterized by calculating the amount of change in spatial resolution before and after parallelization using one of the following methods.

[0034] The amount of change is calculated by using the difference or ratio between the pixel amount for each incident angle before and after parallelization transformation, or the projected pixel amount per unit of the surface of the assumed detection object if the surface can be assumed. More specifically, the difference may be the absolute value of the difference, and the amount of change in the ratio may be the absolute value of the logarithm of the ratio.

[0035] The pixel amount before rectification (or its logarithm) assigned to each pixel after rectification may be used as a reference, based on the pixel amount after rectification. Furthermore, since the pixel amount before rectification calculated in this manner can be used to calculate the corresponding angle of incidence, this information may also be used. Furthermore, since the resolution of the angle of incidence per pixel originally captured cannot be obtained after conversion, the angle of incidence may be corrected if the pixel amount before conversion per pixel after conversion is less than 1. Furthermore, if image interpolation processing is involved during the process, the angle of incidence may be corrected accordingly.

[0036] After estimating the above-described conversion amounts, the collimation optical axis priority determination unit B07 sets high optical axis selection priorities for optical axes with little overall change (small change), optical axes with little change within a particular region of interest in the parallax image if such region exists, and optical axes with little change along a horizontal line of the parallax image, which is the parallax search direction. If optical axis options are prepared in advance, the calculation results and priorities are stored in advance, and a high priority is assigned to an appropriate optical axis according to the stored calculation results and priorities.

[0037] FIG. 6 shows an example configuration of the measurement accuracy estimation unit B10. The measurement accuracy estimation unit B10 includes a measurement error model generation unit F01 that generates a measurement error model representing the parallax measurement error based on the position and shape of the detected object output from the object detection unit B06. The collimation optical axis priority determination unit B07 sets (determines) the priority of the collimation optical axis according to the results of the measurement error model generation unit F01. Alternatively, the collimation optical axis priority determination unit B07 may set the priority for the collimation optical axis using the above information from the pre-set collimation optical axis. That is, the measurement accuracy estimation unit B10 estimates the parallax measurement accuracy when parallax is calculated using an image after collimation processing using multiple optical axis candidates, and the collimation optical axis priority determination unit B07 determines the priority of the collimation optical axis based on the estimated measurement accuracy. Here, the measurement error model can be thought of as a model determined according to the object and direction of imaging, such that, for example, when a situation like that shown in Figure 4 is assumed, in scene D10, the vehicle glass is imaged, which is likely to cause erroneous measurements, and therefore there will be a large amount of error, but in scene D20, the imaging direction will have a small amount of error.

[0038] After estimating the measurement accuracy as described above, the parallelization optical axis priority determination unit B07 sets a high optical axis selection priority for the optical axis candidate that is deemed to have a high measurement accuracy and selects it as the parallelization optical axis.

[0039] 7 shows an example of the configuration of an in-vehicle image processing device according to an embodiment of the present invention, which determines the priority of the collimated optical axis according to the type of vehicle operation state G01 and driving assistance function state G02. Here, the vehicle operation state refers to a state in which the vehicle is moving forward (high speed, medium speed, low speed), reversing, turning right or left, or during camera calibration such as aiming, and the driving assistance function refers to a state in which driving assistance functions such as ACC and lane keeping systems, or calibration and adjustment of the imaging device such as calibration and aiming, are active.

[0040] FIG. 8 shows an example of prioritizing the collimated optical axis (virtual optical axis) depending on the vehicle operating state and the driving assistance function state. In this example, when the vehicle is operating at high or medium speed, a virtual optical axis (H11) that is nearly horizontal to the road surface is selected so that three-dimensional objects at a distance can be easily observed (H12) (Scene H10). In other words, when the vehicle speed exceeds a predetermined value, a high priority is given to an optical axis that is nearly parallel to the road surface around the vehicle among multiple optical axis candidates. On the other hand, when the vehicle is operating at low speed, reversing, parking, or pulling over to the shoulder when turning right or left, requiring careful attention to the vehicle's footing, a virtual optical axis (H21) that is close to the road surface is selected so that areas close to the vehicle's tires and body are captured (Scene H20). Here, the road edge (shoulder) (H13, H23) is used as an example. In other words, when the vehicle speed is below a predetermined value or when the vehicle's driving assistance function is executing parking assistance, the optical axis directed toward the road surface surrounding the vehicle is selected from among the multiple optical axis candidates with a high priority. Also, as the distance to the approaching object becomes closer to the vehicle, the optical axis directed toward the road surface surrounding the vehicle is selected with a high priority. Note that even at low speeds, when entering an intersection, a virtual optical axis is selected in which the parallelized optical axis is nearly horizontal to the road surface (in other words, when it is determined that the vehicle has entered an intersection, the optical axis that is approximately parallel to the road surface surrounding the vehicle is selected from among the multiple optical axis candidates with a high priority), making it possible to switch so as to make it easier to measure three-dimensional objects.

[0041] As another example, when the driving assistance function is calibrating or adjusting the imaging device, such as aiming or calibration, the collimated optical axis can be selected so as to be suitable for capturing an image of a target calibration equipment. In other words, when the driving assistance function of the vehicle calibrates the imaging device, the priority of the optical axis that is suitable for capturing an image of the calibration equipment of the imaging device is set high and selected.

[0042] 9 shows an example configuration of an in-vehicle image processing device according to an embodiment of the present invention, which combines multiple image capture devices. This configuration includes three or more image capture devices, with each pair of image capture devices equipped with a parallelization unit, a parallax calculation unit, and an object detection unit. Specifically, in the in-vehicle image processing device B00, the image acquisition unit B03 includes a first image acquisition unit B031 that acquires images from multiple image capture devices B01 and B02, and a second image acquisition unit B032 that acquires images from multiple image capture devices B02 and B12. The parallelization processing unit B04, the parallax calculation unit B05, and the object detection unit B06 include a first parallelization processing unit B041, a first parallax calculation unit B051, and a first object detection unit B061 that process the images acquired by the first image acquisition unit B031, and a second parallelization processing unit B042, a second parallax calculation unit B052, and a second object detection unit B062 that process the images acquired by the second image acquisition unit B032, respectively. The in-vehicle image processing device B00 also includes a parallelization optical axis arbitration unit I01, which arbitrates between (multiple) parallelization optical axes based on a detection range arbitration unit I02, a spatial resolution arbitration unit I03, and a measurement accuracy arbitration unit I04 to select the optimal parallelization optical axis.

[0043] The collimation optical axis arbitration unit I01 arbitrates among the multiple collimations to optimize the whole and selects a collimation optical axis.

[0044] For example, if the detection range arbitration unit I02 selects a collimated optical axis for the pair of image capture devices B01 and B02 and as a result some areas are not visible, it is conceivable that the detection range arbitration unit I02 may select a collimated optical axis for the pair of image capture devices B02 and B12 to compensate for that area. As a result, the image capture devices B02 and B12 cover the areas that are not visible for the image capture devices B01 and B02, resulting in a configuration in which the detection range is compensated for as a whole.

[0045] Furthermore, if the change in spatial resolution is large in a certain region, the spatial resolution arbitration unit 103 selects a collimation optical axis that minimizes the change in spatial resolution for the region where the change in spatial resolution is deemed large. As a result, a configuration can be realized that minimizes the change in spatial resolution throughout the entire system.

[0046] Similarly, the measurement accuracy arbitration unit I04 selects a parallel optical axis between the cameras in a flexible manner so that the error estimated by the measurement error model becomes small.

[0047] The collimation optical axis arbitration unit 101 sets appropriate values ​​for the optical axes of multiple cameras according to the priorities determined by these arbitration units 102, 103, and 104. Although this example shows an example with three image capture devices, the same thing can be done with a configuration with four or more image capture devices.

[0048] As described above, the in-vehicle image processing device B00 of this embodiment includes an image acquisition unit B03 that acquires images from a plurality of imaging devices, each imaging device being (a plurality of) imaging devices mounted on a vehicle and capturing a plurality of images from different viewpoints; a parallelization optical axis priority determination unit B07 that determines the priority of a plurality of optical axis candidates, which are any optical axes on a plane perpendicular to the baseline direction of the plurality of imaging devices; a parallelization processing unit B04 that parallelizes the plurality of images using at least one optical axis of the optical axis candidate whose priority has been determined, and outputs the plurality of images; a parallax calculation unit B05 that calculates parallax based on the plurality of images output from the parallelization processing unit B04; and an object detection unit B06 that detects an object (within the image) based on the parallax.

[0049] The in-vehicle image processing device B00 further includes a spatial resolution calculation unit B09 that calculates the change in spatial resolution between the image before the parallelization processing and the image after the parallelization processing has been performed using the plurality of optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority based on the change in spatial resolution.

[0050] The in-vehicle image processing device B00 includes a detection range calculation unit B08 that calculates an object detection range occupied by an object to be detected in an image after the parallelization process using the plurality of optical axis candidates, and the parallelized optical axis priority determination unit B07 determines the priority based on the object detection range.

[0051] The in-vehicle image processing device B00 includes a measurement accuracy estimation unit B10 that estimates the measurement accuracy of the parallax when the parallax is calculated using an image after the parallelization process using the plurality of optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority based on the estimated measurement accuracy.

[0052] The parallelized optical axis priority determination unit B07 changes the priority based on the operating state of the vehicle.

[0053] When the speed of the vehicle is below a predetermined value or when the driving assistance function of the vehicle is performing parking assistance, the parallelization optical axis priority determination unit B07 sets a high priority to the optical axis that is directed toward the road surface around the vehicle among the multiple optical axis candidates.

[0054] When the speed of the vehicle exceeds a predetermined value or when it is determined that the vehicle has entered an intersection, the parallelized optical axis priority determination unit B07 sets a high priority to an optical axis that is approximately parallel to the road surface around the vehicle among the multiple optical axis candidates.

[0055] When the driving assistance function of the vehicle calibrates the imaging device, the parallelized optical axis priority determination unit B07 sets a high priority to an optical axis that is suitable for imaging equipment used to calibrate the imaging device.

[0056] In other words, the parallelization optical axis priority determination unit B07 determines candidate directions of the optical axis for parallelizing each image based on information from any of the following: a detection range determination unit (detection range calculation unit B08) that determines whether or not an image of the object or part of the object to be detected can be captured; a spatial resolution calculation unit B09 that calculates the amount of change in spatial resolution before and after parallelization; and a measurement accuracy estimation unit B10 that estimates the measurement accuracy of the object to be measured.

[0057] The detection range determination unit (detection range calculation unit B08) estimates whether or not a specific detection object or part of the detection object can be imaged based on the candidates for the parallelization optical axis, and based on the results, selects the candidate that is deemed to have the widest detection range as the parallelization optical axis.

[0058] The spatial resolution calculation unit B09 calculates the change in spatial resolution before and after parallelization conversion for each imaging device according to the candidates for the parallelization optical axis, and selects the candidate for which the change in spatial resolution for both imaging devices is deemed to be small as a result of the spatial resolution calculation unit B09 as the parallelization axis.

[0059] The measurement accuracy estimation unit B10 estimates the measurement efficiency and measurement accuracy of a predetermined detection object according to the candidates for the collimated optical axis, and selects the candidate that is deemed to have the highest measurement accuracy as the collimated optical axis as a result of the measurement accuracy estimation unit B10. The measurement accuracy estimation unit B10 estimates the surface of the detection object, and selects the optical axis so that it is approximately perpendicular to the surface.

[0060] The parallelization optical axis priority determination unit B07 changes the optical axis priority based on the operating state of the vehicle and whether a driving assistance function is on or off.

[0061] When the driving assistance function is parking assist or the vehicle is traveling at a low speed, the in-vehicle image processing device B00 selects a parallel optical axis so that an object closer to the vehicle is imaged.

[0062] When the vehicle is traveling at high speed, the vehicle-mounted image processing device B00 selects an axis in which the collimated optical axis is substantially parallel to the road surface.

[0063] The vehicle-mounted image processing device B00 selects a parallel optical axis in the direction of the road surface as the distance of an approaching object to the vehicle decreases.

[0064] When the driving support function determines that the vehicle has entered an intersection, the in-vehicle image processing device B00 selects the collimated optical axis so that the collimated optical axis is approximately parallel to the road surface.

[0065] When the driving assistance function calibrates the imaging device, the in-vehicle image processing device B00 selects a collimated optical axis that is suitable for capturing an image of equipment used for calibrating the imaging device.

[0066] When the priorities of the collimation optical axes compete with each other, the collimation optical axis priority determination unit B07 selects a plurality of collimation optical axes and performs a plurality of parallax measurements.

[0067] When there is a conflict between the collimation optical axes, the collimation optical axis priority determination unit B07 evaluates the danger to each object and selects the collimation optical axis that can best detect or measure the object based on the degree of danger.

[0068] According to this embodiment, the selection priority of the collimated optical axis is appropriately set in accordance with various situations to select an appropriate collimated optical axis, and then the collimation process is performed to perform three-dimensional measurement, thereby making it possible to efficiently measure the distance to the target object.

[0069] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0070] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, SD card, or DVD.

[0071] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0072] B00 In-vehicle image processing device B01 Imaging device B02 Imaging device B03 Image acquisition unit B04 Collimation processing unit B05 Parallax calculation unit B06 Object detection unit B07 Collimation optical axis priority determination unit B08 Detection range calculation unit B09 Spatial resolution calculation unit B10 Measurement accuracy estimation unit G01 Vehicle operation state G02 Driving assistance function state I01 Collimation optical axis arbitration unit I02 Detection range arbitration unit I03 Spatial resolution arbitration unit I04 Measurement accuracy arbitration unit

Claims

1. An in-vehicle image processing device comprising: an image acquisition unit that acquires images from a plurality of imaging devices mounted on a vehicle, the image acquisition unit acquiring images from the plurality of imaging devices capturing a plurality of images from different viewpoints; a parallelization optical axis priority determination unit that determines the priority of a plurality of optical axis candidates that are any optical axes on a plane perpendicular to a baseline direction of the plurality of imaging devices; a parallelization processing unit that parallelizes the plurality of images using at least one of the optical axis candidates whose priority has been determined, and outputs the plurality of images; a parallax calculation unit that calculates parallax based on the plurality of images output from the parallelization processing unit; and an object detection unit that detects an object based on the parallax.

2. An in-vehicle image processing device as described in claim 1, further comprising a spatial resolution calculation unit that determines a change in spatial resolution between an image before the parallelization processing and an image after the parallelization processing has been performed using the multiple optical axis candidates, and the parallelization optical axis priority determination unit determines the priority based on the change in spatial resolution.

3. An in-vehicle image processing device as described in claim 1, further comprising a detection range calculation unit which determines an object detection range occupied by an object to be detected in an image after the parallelization process has been performed using the plurality of optical axis candidates, and the parallelization optical axis priority determination unit determines the priority based on the object detection range.

4. An in-vehicle image processing device as described in claim 1, further comprising a measurement accuracy estimation unit that estimates the measurement accuracy of the disparity when the disparity is calculated using an image after the parallelization process is performed using the multiple optical axis candidates, and the parallelization optical axis priority determination unit determines the priority based on the estimated measurement accuracy.

5. An in-vehicle image processing device according to claim 1, wherein the parallelization optical axis priority determination unit changes the priority based on an operating state of the vehicle.

6. An in-vehicle image processing device as described in claim 5, wherein the parallelization optical axis priority determination unit sets a high priority to an optical axis that is directed toward the road surface around the vehicle among the multiple optical axis candidates when the speed of the vehicle is below a predetermined value.

7. An in-vehicle image processing device as described in claim 5, wherein the parallelized optical axis priority determination unit sets a high priority to an optical axis that is directed toward the road surface around the vehicle among the multiple optical axis candidates when the vehicle's driving assistance function is executing parking assistance.

8. An in-vehicle image processing device as described in claim 5, wherein the parallelized optical axis priority determination unit sets a high priority to an optical axis that is approximately parallel to the road surface around the vehicle among the multiple optical axis candidates when the speed of the vehicle exceeds a predetermined value.

9. An in-vehicle image processing device as described in claim 5, wherein the parallelized optical axis priority determination unit sets a high priority to an optical axis that is approximately parallel to the road surface around the vehicle among the multiple optical axis candidates when it is determined that the vehicle has entered an intersection.

10. An in-vehicle image processing device as described in claim 5, wherein the parallelized optical axis priority determination unit sets a high priority for an optical axis that is suitable for imaging equipment for calibrating the imaging device when a driving assistance function of the vehicle calibrates the imaging device.

Images