In-vehicle image processing device

Abstract

To provide an in-vehicle image processing device capable of efficiently measuring a distance to a target object.SOLUTION: An in-vehicle image processing device includes: an image acquisition unit B03 that acquires images from a plurality of imaging devices for capturing a plurality of images having different viewpoints; a parallelization optical axis priority determination unit B07 that determines priority for a plurality of optical axis candidates which are arbitrary optical axes on a plane orthogonal to a base line direction of the plurality of imaging devices; a parallelization processing unit B04 that performs parallelization processing on the plurality of images and outputs the results, by using at least one optical axis among the optical axis candidates for which the priority is determined; a parallax calculation unit B05 that determines parallax on the basis of the plurality of images output from the parallelization processing unit B04; and an object detection unit B06 that detects objects (in the images) on the basis of the parallax.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an in-vehicle image processing device suitable for a system that has a high level of understanding of the surrounding environment in an automated driving or advanced safe driving support system. [Background technology]

[0002] In order to realize autonomous driving and advanced safety driving support systems, cameras that monitor the outside world and detect objects necessary for the vehicle's navigation, such as obstacles and lane information, are becoming increasingly important.

[0003] In particular, to improve detection performance, multiple cameras are installed on the vehicle to obtain information about the vehicle's surroundings and realize an external environment recognition function.

[0004] There are two types of cameras that recognize the outside world: monocular cameras and stereo cameras that use multiple cameras. Stereo cameras can measure the distance to an object by using the parallax of the overlapping area between two images taken with two cameras at a certain distance from each other and with different viewpoints.

[0005] Stereo cameras use multiple cameras to measure distances, making it possible to accurately grasp the risk of collision with an object ahead. Distance is generally calculated by connecting multiple cameras to a single computing resource such as a System on the Chip (SoC), aggregating images, and deriving the correspondence between images while accessing them to determine the distance.

[0006] When measuring the distance to an object using the principle of a stereo camera, the process of transforming the input images so that objects in two images are aligned on the same line is called parallelization. Parallelization requires image distortion correction and image rotation and projective transformation. In this parallelization process, it is necessary to determine the optical axis after parallelization, and this can be any axis that is perpendicular to the base line connecting the positions of the two cameras, as long as it includes an overlapping area. For example, Patent Document 1 discloses a technology for setting the optical axis after parallelization in a forward-facing stereo camera so that it faces the road surface. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2019-16308 A Summary of the Invention [Problem to be solved by the invention]

[0008] Three-dimensional measurements based on the principle of stereo cameras can be performed not only with cameras placed in front, but also with stereo cameras placed in the periphery. Stereo cameras placed in the periphery have a high degree of freedom, such as combining images with large distortions or having an oblique baseline, and depending on the optical axis settings, this can affect detection capabilities and measurement accuracy, resulting in a decrease in measurement performance.

[0009] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an in-vehicle image processing device that can efficiently measure the distance to an object. [Means for solving the problem]

[0010] In order to solve the above problems, the in-vehicle image processing device of the present invention is an imaging device mounted on a vehicle, and includes an image acquisition unit that acquires images from a plurality of imaging devices that capture a plurality of images from different viewpoints, a parallelization optical axis priority determination unit that determines the priority of a plurality of optical axis candidates that are any optical axes on a plane perpendicular to the baseline direction of the plurality of imaging devices, a parallelization processing unit that parallelizes the plurality of images using at least one of the optical axis candidates whose priority has been determined, and outputs the images, a parallax calculation unit that calculates parallax based on the plurality of images output from the parallelization processing unit, and an object detection unit that detects an object based on the parallax. Effect of the Invention

[0011] According to the present invention, the distance to an object can be measured efficiently.

[0012] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram illustrating a multi-camera system and a coordinate system according to an embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating a configuration of an in-vehicle image processing device according to an embodiment of the present invention. [Diagram 3] FIG. 4 is a diagram for explaining the configuration of a detection range calculation unit. [Figure 4] 11A and 11B are diagrams for explaining examples of use of a detection range calculation unit. [Diagram 5] FIG. 4 is a diagram for explaining the configuration of a spatial resolution calculation unit. [Figure 6] FIG. 4 is a diagram for explaining the configuration of a measurement accuracy estimation unit. [Figure 7] 1 is a diagram for explaining the configuration of an in-vehicle image processing device according to an embodiment of the present invention, which has a vehicle operation state and a driving assistance function state. [Figure 8] 11A to 11C are diagrams for explaining examples of utilization of a detection range calculation unit according to an operation mode. [Figure 9]1 is a diagram illustrating the configuration of an in-vehicle image processing device according to an embodiment of the present invention, in which a plurality of imaging devices are combined. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing for explaining the embodiment of the present invention, parts having the same functions are given the same reference numerals, and repeated description thereof will be omitted.

[0015] FIG. 1 shows the configuration of a multi-camera system A006 in which a front camera A001, a left front camera A002, a right front camera A003, a left side camera A004, and a right side camera A005 are attached to the front, left front, right front, left side, and right side of a vehicle A000, respectively. The multi-camera system A006 composed of these cameras estimates and acquires information on surrounding objects such as pedestrians, vehicles, white lines, and road surfaces, as well as physical quantities such as their distances, sizes, and speeds, and determines the vehicle control amount based on the information to perform automatic driving, driving assistance, and the like. The vehicle A000 is equipped with an in-vehicle image processing device B00 (FIG. 2) for realizing the above-mentioned processing. In this embodiment, a vehicle coordinate system (world coordinate system) is taken as shown in A007, and the traveling direction of the vehicle A000 is taken as the X-axis, the horizontal direction is taken as the Y-axis, and the vertical direction is taken as the Z-axis. The image coordinate system is taken as shown in A008.

[0016] There are several methods for estimating the distance to an object, but a typical example is when multiple cameras capture images of the same object, using the principle of triangulation to measure the distance.

[0017] Here, any point on the road in the world coordinate system is expressed as X=(X, Y, Z, 1) in the homogeneous coordinate system. T The external parameter matrices related to the camera's rotation angle and installation position are R and T, respectively. The matrix is ​​P = (R | T). The internal parameter matrix that manages the internal state of the camera, such as the focal length and optical center, is K. The image coordinates where X was captured are expressed in the homogeneous coordinate system as u = (u, v, 1). TIf the scale parameter is s and the lens projection model is L, the following formula 1 holds for each camera. Note that the subscripts indicate the type of camera. [Number 1] TIFF2025016107000002.tif15147

[0018] Disparity measurement using stereo processing is possible by finding corresponding points in these two images, but by performing a process called rectification, it is possible to convert the images so that identical points are aligned on the same line on the image. This rectification of the two cameras is achieved by solving the following equation 2. [Number 2] TIFF2025016107000003.tif85160

[0019] Here, there is a degree of freedom in how T' is chosen, and basically any direction is acceptable as long as it is a vector perpendicular to the baseline connecting the two cameras. The way T' is chosen is directly related to how the image after collimation is generated. Below, this is called the collimation optical axis.

[0020] Depending on how the parallel optical axis is set, an image with a significantly different resolution ratio from the original may be created, which may result in a deterioration in the accuracy of the calculation of parallax, etc. Also, it may change whether the object to be detected can be observed or not.

[0021] The image coordinates after rectification are expressed as u rect Let L be the projection system onto the rectified image. rect Then, each image coordinate can be expressed by the following relational expression 3. [Number 3] TIFF2025016107000004.tif36148

[0022] 2 shows an example of the configuration of an in-vehicle image processing device according to an embodiment of the present invention. The in-vehicle image processing device B00 of this embodiment is composed of an image acquisition unit B03, a parallelization processing unit B04, a parallax calculation unit B05, an object detection unit B06, and a parallelization optical axis priority determination unit B07, and the parallelization optical axis priority determination unit B07 is characterized in that it determines the priority of the parallelization optical axis based on a detection range calculation unit B08, a spatial resolution calculation unit B09, a measurement accuracy estimation unit B10, and a camera parameter group B11 including the camera parameters shown in Equations 1 to 3 and the size and processing range of the image.

[0023] The image acquisition unit B03 acquires images from a plurality of imaging devices B01, B02 mounted on a vehicle, which capture a plurality of images from different viewpoints, and transmits the acquired images to a parallelization processing unit B04. As an example, the plurality of imaging devices B01, B02 are configured from a pair of cameras (imaging devices) that share a part of a field of view or have a part of a field of view that overlaps (has an overlapping area) among the front camera A001, the left front camera A002, the right front camera A003, the left side camera A004, and the right side camera A005 shown in FIG. 1.

[0024] The parallelization processing unit B04 parallelizes the images obtained from the two imaging devices B01, B02 based on the image information transmitted from the image acquisition unit B03, the camera parameter group B11 updated in advance or during driving, and the optical axis selected by the parallelization optical axis priority determination unit B07, and transmits the parallelization results to the downstream parallax calculation unit B05.

[0025] The disparity calculation unit B05 generates a disparity image based on the multiple images rectified by the rectification processing unit B04 using techniques such as block matching, feature point matching, and deep learning, and transmits the generated image to the downstream object detection unit B06.

[0026] The object detection unit B06 analyzes the parallax image generated by the parallax calculation unit B05 to detect an object.

[0027] The parallelization optical axis priority determination unit B07 sets (determines) priorities for the parallelization optical axes (multiple optical axis candidate positions which are any optical axes on a plane perpendicular to the baseline direction of the multiple imaging devices B01, B02) based on the results of the detection range calculation unit B08, the spatial resolution calculation unit B09, and the measurement accuracy estimation unit B10, and selects, for example, the parallelization optical axis with a high priority and transmits it to the parallelization processing unit B04.

[0028] FIG. 3 shows an example of the configuration of the detection range calculation unit B08. The detection range calculation unit B08 includes a detection target range determination unit C01 that outputs the position of an expected target object, and a detection range estimation unit C03 that determines (estimates) whether the position of the target object output by the detection target range determination unit C01 is within the angle of view of the parallax image (object detection range) using information C02, particularly the size and range of the parallax image, from the camera parameter group B11. The parallelization optical axis priority determination unit B07 is characterized in that it sets (determines) and selects a parallelization optical axis suitable for satisfying the criterion as having a high priority. That is, the detection range calculation unit B08 obtains an object detection range occupied by an object to be detected in an image after parallelization processing using a plurality of optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority of the parallelization optical axis based on the object detection range. The parallelization optical axis may be set from the parallelization optical axis set in advance using the above criterion, or an optical axis satisfying the criterion may be analytically selected. The detection object and its range used by detection object range determination unit C01 are determined based on the position and range of the object detected by object detection unit B06 and the result of the position and range of the object estimated in time series.

[0029] For example, FIG. 4 shows an example of a scene where a vehicle runs parallel to the host vehicle. In the scene D10, when the parallel optical axis is set as D11, the estimated detection range is set as D12, and only the upper part of the vehicle D13 running parallel to the host vehicle D14 is detected. On the other hand, in the scene D20, when the parallel optical axis is set as D21, the estimated detection range is obtained as D22, and the lower part of the vehicle D23 running parallel to the host vehicle D24 is detected at the center. Depending on each detection situation, for example, if the optical axis is once directed in a case such as D10, the optical axis may be changed to D20 when the lower part of the vehicle is to be accurately measured. Such a case is effective, for example, when it is desired to know the exact distance to a vehicle approaching from the side. Specifically, in the case where the object detection unit B06 detects a vehicle D23 running parallel to the vehicle, the detection target range determination unit C01 estimates the overall image from the partial visibility of the vehicle, and the detection range estimation unit C03 determines that the lower part of the vehicle is not visible, and sets (determines) and selects the parallelized optical axis that can capture the image of the vehicle (has a wide object detection range) as having a high priority. The parallelization processing unit B04, the parallax calculation unit B05, and the object detection unit B06 perform parallelization processing according to the newly set optical axis in this way and measure the distance.

[0030] 5 shows an example of the configuration of the spatial resolution calculation unit B09. The spatial resolution calculation unit B09 includes a pre-parallelization spatial resolution calculation unit E01 indicating the spatial resolution before parallelization, a post-parallelization spatial resolution calculation unit E02 indicating the spatial resolution after parallelization, and a spatial resolution change amount calculation unit E03 calculating the difference in spatial resolution before and after parallelization using information of the camera parameter group B11, and the parallelization optical axis priority determination unit B07 sets (determines) a collimation optical axis that is suitable according to the result of the spatial resolution change amount calculation unit E03 as having a high priority. Alternatively, the parallelization optical axis priority determination unit B07 may set a priority for the collimation optical axis using the above information from the collimation optical axis set in advance. That is, the spatial resolution calculation unit B09 obtains a change in spatial resolution between an image before parallelization processing and an image after parallelization processing is performed using a plurality of optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority for the collimation optical axis based on the change in spatial resolution.

[0031] The pre-collimation resolution is estimated from internal parameters such as the lens distortion model, focal length, and image size. Here, the pre-collimation resolution may simply be returned as it is in units of one pixel without specifying the angle of incidence, for example. The magnitude of the angle of incidence per pixel may also be considered. The size of the imaging surface captured per pixel may also be calculated by assuming the surface of the object to be detected using the external parameters of the camera.

[0032] The resolution after collimation is estimated from the internal parameters after collimation, such as the collimated optical axis, the projection model at the time of collimation, the focal length, and the image size, and can be the magnitude of the incident angle per pixel, as in the case of the resolution before collimation. Alternatively, the pixel amount per incident angle, which is the reciprocal of the incident angle, may be used as the basis. When the external parameters of the camera are used, the size of the imaging surface captured per pixel may be calculated by assuming the surface of the detection target.

[0033] The spatial resolution change amount calculation unit E03 is characterized in that it calculates the amount of change in spatial resolution before and after parallelization using any one of the following methods.

[0034] The amount of pixels for each angle of incidence before and after parallelization, or the amount of projected pixels per unit of the surface of the assumed detection object if it can be assumed, is used to calculate the amount of change by taking the difference or ratio between them at the same angle of incidence. More specifically, the difference may be the absolute value of the difference, and the amount of change in the ratio may be the absolute value of the logarithm of the ratio.

[0035] The pixel amount before parallelization (or its logarithm) assigned to each pixel after parallelization may be used with the pixel after parallelization as a reference. Also, since the pixel amount before parallelization calculated in this way can be used to calculate the corresponding angle of incidence, this information may be used. Also, since the resolution of the angle of incidence per pixel originally captured cannot be obtained after conversion, the angle of incidence may be corrected if the pixel amount before conversion per pixel after conversion is less than 1. Also, if image complementation processing is involved midway, the angle of incidence may be corrected accordingly.

[0036] After estimating the above conversion amount, the parallelization optical axis priority determination unit B07 sets high optical axis selection priority for optical axes with little overall change (small change), optical axes with little change within a particular area of ​​interest in the parallax image if such an area exists, and optical axes with little change in a horizontal line of the parallax image which is the parallax search direction.When optical axis options are prepared in advance, the above calculation results and priorities are stored in advance, and a high priority is assigned to an appropriate optical axis according to the stored calculation results and priorities.

[0037] FIG. 6 shows a configuration example of the measurement accuracy estimation unit B10. The measurement accuracy estimation unit B10 includes a measurement error model generation unit F01 that generates a measurement error model representing a measurement error of the parallax based on the position and shape of the detected object output from the object detection unit B06, and the parallelization optical axis priority determination unit B07 sets (determines) the priority of the parallelization optical axis according to the result of the measurement error model generation unit F01. Alternatively, the parallelization optical axis priority determination unit B07 may set the priority for the parallelization optical axis using the above information from the previously set parallelization optical axis. That is, the measurement accuracy estimation unit B10 estimates the measurement accuracy of the parallax when the parallax is obtained using an image after performing a parallelization process using a plurality of optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority of the parallelization optical axis based on the estimated measurement accuracy. Here, the measurement error model can be thought of as having a model that is determined according to the imaged object and imaging direction, for example, assuming a situation like that shown in Figure 4, in scene D10, the vehicle glass is captured, which is expected to lead to a situation that is prone to erroneous measurements, and therefore there will be a lot of error, but in the case of the imaging direction like scene D20, there will be less error.

[0038] After estimating the measurement accuracy as described above, the collimation optical axis priority determination unit B07 sets the optical axis selection priority to be high for the optical axis candidate that is deemed to have a high measurement accuracy, and selects it as the collimation optical axis.

[0039] 7 shows an example of the configuration of an in-vehicle image processing device according to an embodiment of the present invention, which determines the priority of the parallelized optical axis according to the types of vehicle operation state G01 and driving support function state G02. Here, the vehicle operation state refers to a state in which the vehicle is moving forward (high speed, medium speed, low speed), moving backward, turning right or left, or when the camera is calibrated, such as during aiming, and the driving support function refers to a state in which the driving support function, such as the ACC or lane keeping system, or the calibration or adjustment of the imaging device, such as calibration or aiming, is working.

[0040] FIG. 8 shows an example of dividing the priority of the parallelized optical axis (virtual optical axis) according to the vehicle operation state and the driving assistance function state. In this example, when the vehicle operation state is high or medium speed, a virtual optical axis (H11) that makes the parallelized optical axis nearly horizontal to the road surface is selected so that a solid object somewhat far away can be easily observed (H12) (Scene H10). In other words, when the vehicle speed exceeds a predetermined value, the priority of an optical axis that is nearly parallel to the road surface around the vehicle is set high and selected from among multiple optical axis candidates. On the other hand, when the vehicle operation state is low speed, backing up, parking, or pulling over to the shoulder when turning right or left, and driving while paying attention to the vehicle's feet is required, a virtual optical axis (H21) that makes the parallelized optical axis approach the road surface is selected so that a place close to the tires or body of the vehicle is imaged (H22) (Scene H20). Here, the road edge (shoulder) (H13, H23) is taken as an example. In other words, when the vehicle speed is below a predetermined value or when the vehicle's driving assistance function is executing parking assistance, the priority of the optical axis directed toward the road surface surrounding the vehicle is set high and selected from among the multiple optical axis candidates. Also, as the distance of the approaching object to the vehicle decreases, the priority of the optical axis directed toward the road surface surrounding the vehicle is set high and selected. Note that even at low speeds, when entering an intersection, a virtual optical axis is selected in which the parallelized optical axis is nearly horizontal to the road surface (in other words, when it is determined that the vehicle has entered an intersection, the priority of the optical axis approximately parallel to the road surface surrounding the vehicle is set high and selected from among the multiple optical axis candidates), making it possible to switch so as to make it easier to measure a three-dimensional object.

[0041] As another example, when the driving assistance function is calibrating or adjusting the imaging device such as aiming or calibration, the collimated optical axis can be selected so as to be suitable for imaging the target calibration equipment. In other words, when the driving assistance function of the vehicle calibrates the imaging device, the priority of the optical axis suitable for imaging the calibration equipment of the imaging device is set high and selected.

[0042] 9 shows an example of the configuration of an in-vehicle image processing device according to an embodiment of the present invention, in which a plurality of imaging devices are combined. In this configuration, three or more imaging devices are provided, and a parallelization means, a parallax calculation means, and an object detection means are provided for each pair of imaging devices. That is, in the in-vehicle image processing device B00, the image acquisition unit B03 has a first image acquisition unit B031 that acquires images from a plurality of imaging devices B01 and B02, and a second image acquisition unit B032 that acquires images from a plurality of imaging devices B02 and B12, and the parallelization processing unit B04, the parallax calculation unit B05, and the object detection unit B06 have a first parallelization processing unit B041, a first parallax calculation unit B051, and a first object detection unit B061 that process images acquired by the first image acquisition unit B031, and a second parallelization processing unit B042, a second parallax calculation unit B052, and a second object detection unit B062 that process images acquired by the second image acquisition unit B032, respectively. The in-vehicle image processing device B00 also includes a parallelization optical axis arbitration unit I01, which arbitrates between (multiple) parallelization optical axes based on a detection range arbitration unit I02, a spatial resolution arbitration unit I03, and a measurement accuracy arbitration unit I04 to select an optimal parallelization optical axis.

[0043] The collimation optical axis arbitration unit I01 arbitrates among the multiple collimations so as to optimize the whole and selects a collimation optical axis.

[0044] For example, if the detection range arbitration unit I02 selects a certain collimated optical axis for the pair of imaging device B01 and imaging device B02 and as a result some area is not visible, it is possible to select the collimated optical axis for the pair of imaging device B02 and imaging device B12 to complement that area. As a result, the imaging device B02 and imaging device B12 cover the area where the imaging device B01 and imaging device B02 are not visible, resulting in a configuration that complements the detection range as a whole.

[0045] In addition, if the spatial resolution change is large in a part of the region, the spatial resolution arbitration unit I03 selects a collimation optical axis that minimizes the change in spatial resolution for the region where the change in spatial resolution is deemed to be large. As a result, a configuration that covers the entire system so that the change in spatial resolution is minimized can be realized.

[0046] Similarly, the measurement accuracy arbitration unit I04 selects a parallelizing optical axis among the cameras in a flexible manner so that the error estimated by the measurement error model becomes small.

[0047] According to the priorities determined by these arbitration units I02, I03, and I04, the parallelization optical axis arbitration unit I01 sets appropriate values ​​for the optical axes of multiple cameras. Although three imaging devices are shown as an example in this example, the same thing can be done with a configuration equipped with four or more imaging devices.

[0048] As described above, the in-vehicle image processing device B00 of this embodiment includes an image acquisition unit B03 that acquires images from multiple imaging devices, which are (multiple) imaging devices mounted on a vehicle and capture multiple images from different viewpoints, a parallelization optical axis priority determination unit B07 that determines the priority of multiple optical axis candidates, which are any optical axes on a plane perpendicular to the baseline direction of the multiple imaging devices, a parallelization processing unit B04 that parallelizes the multiple images using at least one optical axis among the optical axis candidates whose priority has been determined, and outputs the multiple images, a parallax calculation unit B05 that calculates parallax based on the multiple images output from the parallelization processing unit B04, and an object detection unit B06 that detects an object (in the images) based on the parallax.

[0049] The vehicle-mounted image processing device B00 further includes a spatial resolution calculation unit B09 that determines a change in spatial resolution between an image before the parallelization processing and an image after the parallelization processing is performed using the multiple optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority based on the change in spatial resolution.

[0050] The vehicle-mounted image processing device B00 includes a detection range calculation unit B08 that calculates an object detection range occupied by an object to be detected in an image after the parallelization process is performed using the multiple optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority based on the object detection range.

[0051] The in-vehicle image processing device B00 includes a measurement accuracy estimation unit B10 that estimates the measurement accuracy of the disparity when the disparity is calculated using an image after the parallelization process is performed using the multiple optical axis candidates, and the parallelization optical axis priority determination unit B07 determines the priority based on the estimated measurement accuracy.

[0052] The parallelization optical axis priority determination unit B07 changes the priority based on the motion state of the vehicle.

[0053] When the speed of the vehicle is below a predetermined value or when the driving assistance function of the vehicle is executing parking assistance, the parallelization optical axis priority determination unit B07 sets a high priority to an optical axis that is directed toward the road surface around the vehicle among the multiple optical axis candidates.

[0054] When the speed of the vehicle exceeds a predetermined value or when it is determined that the vehicle has entered an intersection, the parallelization optical axis priority determination unit B07 sets a high priority to an optical axis that is approximately parallel to the road surface around the vehicle among the multiple optical axis candidates.

[0055] When a driving assistance function of the vehicle calibrates the imaging device, the parallelization optical axis priority determination unit B07 sets a high priority to an optical axis that is suitable for imaging equipment for calibrating the imaging device.

[0056] In other words, the parallelization optical axis priority determination unit B07 determines candidates for the direction of the optical axis for parallelizing each image based on information from any one of the detection range determination unit (detection range calculation unit B08), which determines whether or not the object or part of the object to be detected can be captured, the spatial resolution calculation unit B09, which calculates the amount of change in spatial resolution before and after parallelization, and the measurement accuracy estimation unit B10, which estimates the measurement accuracy of the object to be measured.

[0057] The detection range determination unit (detection range calculation unit B08) estimates whether or not a specific detection object or part of a detection object can be imaged according to the candidates for the parallelization optical axis, and selects the candidate that is deemed to have the widest detection range based on the result as the parallelization optical axis.

[0058] The spatial resolution calculation unit B09 calculates the change in spatial resolution before and after parallelization conversion for each imaging device according to the candidates for the parallelization optical axis, and selects the candidate for which the change in spatial resolution for both imaging devices is deemed to be small as a result of the spatial resolution calculation unit B09 as the parallelization axis.

[0059] The measurement accuracy estimation unit B10 estimates the measurement efficiency and measurement accuracy of a predetermined detection object according to the candidates for the collimated optical axis, and selects the candidate that is deemed to have the highest measurement accuracy as the collimated optical axis as a result of the measurement accuracy estimation unit B10. The measurement accuracy estimation unit B10 estimates the surface of the detection object, and selects the optical axis so that it is approximately perpendicular to the surface.

[0060] The parallelization optical axis priority determination unit B07 changes optical axis priority based on the operating state of the vehicle and on / off of a driving support function.

[0061] When the driving assistance function is parking assist or the vehicle is moving at a low speed, the in-vehicle image processing device B00 selects a parallel optical axis so that an object closer to the vehicle is imaged.

[0062] When the vehicle is moving at high speed, the vehicle-mounted image processing device B00 selects an axis in which the collimated optical axis is substantially parallel to the road surface.

[0063] The vehicle-mounted image processing device B00 selects a parallel optical axis in the direction of the road surface as the distance of an approaching object to the vehicle becomes shorter.

[0064] When the driving support function determines that the vehicle has entered an intersection, the vehicle-mounted image processing device B00 selects the collimated optical axis so that the collimated optical axis is approximately parallel to the road surface.

[0065] When the driving assistance function calibrates the imaging device, the in-vehicle image processing device B00 selects a collimated optical axis that is suitable for capturing an image of a calibration material for the imaging device.

[0066] When the priorities of the collimation optical axes compete with each other, the collimation optical axis priority determination unit B07 selects a plurality of collimation optical axes and performs a plurality of parallax measurements.

[0067] When there is a competition between the collimation optical axes, the collimation optical axis priority determination unit B07 evaluates the danger to each object, and selects the collimation optical axis that can best detect or measure the object based on the degree of danger.

[0068] According to this embodiment, the selection priority of the collimation optical axis is appropriately set in accordance with various situations to select an appropriate collimation optical axis, and then the collimation process is performed to perform three-dimensional measurement, thereby making it possible to efficiently measure the distance to an object.

[0069] The present invention is not limited to the above-described embodiment, and includes various modified forms. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0070] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0071] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0072] B00 Vehicle-mounted image processing device B01 Imaging device B02 Imaging device B03 Image acquisition unit B04 Parallelization processing section B05 Parallax calculation section B06 Object detection unit B07 Parallelizing optical axis priority determination section B08 Detection range calculation unit B09 Spatial resolution calculation section B10 Measurement accuracy estimation unit G01 Vehicle operation status G02 Driving support function status I01 Parallelizing optical axis arbitration section I02 Detection range arbitration unit I03 Spatial resolution arbitration section I04 Measurement Accuracy Arbitration Department

Claims

1. an image acquisition unit that acquires images from a plurality of imaging devices mounted on a vehicle, the plurality of imaging devices capturing a plurality of images from different viewpoints; a parallelization optical axis priority determination unit that determines priorities of a plurality of optical axis candidates, which are arbitrary optical axes on a plane perpendicular to a baseline direction of the plurality of image pickup devices; a parallelization processing unit that performs parallelization processing on the plurality of images using at least one of the optical axis candidates whose priorities have been determined, and outputs the images; a parallax calculation unit that calculates a parallax based on a plurality of images output from the parallelization processing unit; an object detection unit that detects an object based on the parallax; An in-vehicle image processing device comprising:

2. The vehicle-mounted image processing device according to claim 1, a spatial resolution calculation unit that calculates a change in spatial resolution between an image before the parallelization process and an image after the parallelization process is performed using the plurality of optical axis candidates, The collimation optical axis priority determination unit determines the priority based on a change in the spatial resolution.

3. The vehicle-mounted image processing device according to claim 1, a detection range calculation unit that calculates an object detection range occupied by an object to be detected in an image after the parallelization process is performed using the plurality of optical axis candidates; The collimated optical axis priority determination unit determines the priority based on the object detection range.

4. The vehicle-mounted image processing device according to claim 1, a measurement accuracy estimating unit that estimates a measurement accuracy of the parallax when the parallax is calculated using an image after the parallelization process is performed using the plurality of optical axis candidates; The collimation optical axis priority determination unit determines the priority based on the estimated measurement accuracy.

5. The vehicle-mounted image processing device according to claim 1, The parallelization optical axis priority determination unit changes the priority based on an operating state of the vehicle.

6. 6. The vehicle-mounted image processing device according to claim 5, The parallelization optical axis priority determination unit sets a high priority to an optical axis that is directed toward a road surface around the vehicle among the plurality of optical axis candidates when the speed of the vehicle is below a predetermined value.

7. 6. The vehicle-mounted image processing device according to claim 5, The parallelized optical axis priority determination unit sets a high priority to an optical axis directed toward a road surface surrounding the vehicle among the multiple optical axis candidates when a driving assistance function of the vehicle is performing parking assist.

8. 6. The vehicle-mounted image processing device according to claim 5, The parallelized optical axis priority determination unit sets a high priority to an optical axis that is approximately parallel to a road surface around the vehicle among the multiple optical axis candidates when the speed of the vehicle exceeds a predetermined value.

9. 6. The vehicle-mounted image processing device according to claim 5, The parallelized optical axis priority determination unit sets a high priority to an optical axis that is approximately parallel to a road surface around the vehicle among the multiple optical axis candidates when it is determined that the vehicle has entered an intersection.

10. 6. The vehicle-mounted image processing device according to claim 5, The parallelized optical axis priority determination unit sets a high priority to an optical axis that is suitable for imaging equipment for calibrating the imaging device when a driving assistance function of the vehicle calibrates the imaging device.

Images