Image processing device and image processing method

Abstract

To select, from a plurality of distance measuring means, one with a small distance measurement error to reduce a distance measurement error.SOLUTION: An image processing apparatus comprises: an image acquisition unit that acquires a plurality of images picked up by a plurality of cameras; a first distance estimation unit that estimates the distance to an object by using the plurality of images; a second distance estimation unit that estimates the distance to the object by using at least one of the plurality of images; an imaging time calculation unit that determines the imaging times of at least some pixels included in the plurality of images; a reliability calculation unit that determines the reliability of the plurality of images based on the difference between the imaging times of the pixels in the plurality of images; and a distance determination unit that determines the distance to the object by using the reliability and the distance estimated by at least either one of the first distance estimation unit or the second distance estimation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] With the spread of in-vehicle camera systems, which are image recognition devices that recognize objects based on captured images, there is an increasing demand for various recognition functions aimed at safer and more autonomous driving. Among these, stereo camera systems, which detect objects using two cameras arranged on the left and right, simultaneously measure visual information from images and distance information to the object, allowing for detailed understanding of various objects around the vehicle (people, cars, three-dimensional objects, road surfaces, road signs, signboards, etc.), thereby contributing to improved safety during driving assistance.

[0003] A stereo camera calculates distance from the parallax, which is the difference between the positions of the same object captured by the left and right cameras. If the distance to the object is far, the parallax will be small, and if the distance to the object is close, the parallax will be large.

[0004] Stereo cameras require that the cameras be synchronized when capturing images. When the cameras are out of sync, i.e., when one camera experiences a time delay, the other camera captures the image after moving by the time delay. As a result, the projection position of the object on the image differs compared to when two cameras capture images simultaneously, resulting in an incorrect parallax and an incorrect estimation of the distance to the object. For example, if a vehicle captures an object on the side of the road while traveling, the vehicle may use the incorrectly estimated distance to determine whether to avoid a collision, potentially resulting in a collision. In particular, cameras that capture images with a rolling shutter capture each line of the image sequentially, which creates a time delay between lines, resulting in a significant shift in the projection position of the object due to the time delay.

[0005] The following prior art exists as background technology in this technical field: Patent Document 1 (JP 2019-62255 A) describes a stereo image processing device that includes a stereo imaging unit that captures multiple images from different viewpoints, a synchronization unit that synchronizes the imaging times when capturing the multiple images, an image correction unit that parallelizes and outputs the multiple images captured by the stereo imaging unit, a parallax measurement unit that detects parallax based on the images output by the image correction unit, and an object detection unit that detects an object based on the parallax measured by the parallax measurement unit, a delay amount holding unit that holds, for each pixel, a delay time from the imaging time synchronized by the synchronization unit, and an object parallax correction unit that corrects the parallax of an object detected by the object detection unit based on the delay time held by the delay amount holding unit. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-62255 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology disclosed in Patent Document 1 employs an approach to correct parallax error caused by the difference in image capturing time between the left and right cameras of the same object area in a rolling shutter stereo camera, and does not select one of multiple distance measurement means with the smallest distance measurement error. Therefore, the background art does not intend to switch between compound eye distance measurement and single eye distance measurement to use the most suitable one, and is difficult to apply to configurations with three or more cameras or configurations with distance measurement devices other than cameras, resulting in low scalability.

[0008] The present invention has been made in consideration of the above-mentioned circumstances, and aims to select a distance measurement means with small distance measurement error, thereby reducing errors in estimated distance that arise due to differences in the capture times of multiple images. [Means for solving the problem]

[0009] A representative example of the invention disclosed in the present application is as follows: That is, an image processing device including an image acquisition unit that acquires multiple images captured by multiple cameras, a first distance estimation unit that estimates a distance to an object using the multiple images, a second distance estimation unit that estimates a distance to the object using at least one of the multiple images, an image capture time calculation unit that calculates image capture times of at least some pixels included in the multiple images, a reliability calculation unit that calculates reliability of the multiple images based on a difference between image capture times of the pixels in the multiple images, and a distance determination unit that determines a distance to the object using the distance estimated by at least one of the first distance estimation unit or the second distance estimation unit and the reliability. [Effects of the Invention]

[0010] According to one aspect of the present invention, the distance to an object can be estimated with high accuracy. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of an image processing apparatus according to a first embodiment. [Figure 2] 10 is a flowchart of a distance determination process executed by the image processing apparatus according to the first embodiment. [Figure 3] FIG. 10 is a schematic configuration diagram of an image processing apparatus according to a second embodiment. [Figure 4] 10 is a flowchart of a distance determination process executed by an image processing apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, the embodiments will be described with reference to the drawings.

[0013] Example 1 FIG. 1 is a schematic diagram of an image processing device 1 according to a first embodiment of the present invention. The image processing device 1 illustrated in FIG. 1 is a computer that estimates the distance to an object using a camera installed on a moving object such as a vehicle. The image processing device 1 functions as image acquisition units 101a and 101b, a first distance estimation unit 102, a second distance estimation unit 103a and 103b, image capture time calculation units 104a and 104b, a reliability calculation unit 105, and a distance determination unit 106 by executing a program stored in a storage medium (not shown). FIG. 1 illustrates an example in which the image processing device 1 has two image acquisition units 101a and 101b, and accordingly, two second distance estimation units 103a and 103b and two image capture time calculation units 104a and 104b are also provided.

[0014] Image acquisition unit 101a acquires images from an in-vehicle camera or the like and passes them to first distance estimation unit 102, second distance estimation unit 103a, and imaging time calculation unit 104a. Image acquisition unit 101b acquires images from an in-vehicle camera or the like and passes them to first distance estimation unit 102, second distance estimation unit 103b, and imaging time calculation unit 104b.

[0015] The first distance estimation unit 102 estimates the distance to the object using a pair of images of the object acquired by the image acquisition unit 101a and images of the object acquired by the image acquisition unit 101b. Note that known techniques can be used to estimate the distance to the object using images acquired from the multiple image acquisition units 101a and 101b. For example, the distance can be estimated using the parallax of the same object in the two images.

[0016] The second distance estimation unit 103a estimates the distance to the object using an image of the object acquired from the corresponding image acquisition unit 101a. Similarly, the second distance estimation unit 103b estimates the distance to the object using an image of the object acquired from the corresponding image acquisition unit 101b. Note that known techniques can be used to estimate the distance to the object using an image acquired from a single image acquisition unit. For example, in the case of a monocular camera, time-series motion analysis or camera geometric analysis can be used. The second distance estimation units 103a and 103b may measure the distance to the object using a method that does not use an image, i.e., measure the distance based on data acquired by a sensor other than a camera.

[0017] The image capturing time calculation units 104a and 104b calculate the image capturing time of each pixel on the image by using the image capturing start time, exposure time, and image resolution of the images acquired from the image acquisition units 101a and 101b. For example, the image capturing start time is t0, the exposure time is t e If the resolution of an image is (w, h), the image capturing time t of a pixel at coordinates (x, y) in the case of a rolling shutter camera image can be calculated by the following formula.

[0018]

number

[0019] Exposure time t e When is constant, the image capturing time t is determined by the position of the pixel in the image, so the image capturing time t may be calculated only from the pixel position.

[0020] In addition, in the case of a global shutter type camera image, the image capturing time t can be calculated by the following formula, independent of the pixel coordinates.

[0021]

number

[0022] The reliability calculation unit 105 calculates the reliability of a set of pixels recognized as the same object from the difference in the imaging times of pixels recognized as the same object in multiple images. The reliability is defined so that it increases as the difference in imaging time between pixels recognized as the same object decreases, and is maximum when the difference is 0. For example, when the imaging times of pixels recognized as the same object are t1 and t2, respectively, the reliability R is defined by the following equation.

[0023]

number

[0024] Here, k1 and α1 are positive weighting coefficients that should be determined in advance based on the image and distance estimation results. The weighting coefficients k1 and α1 may be predetermined values, but may also be changed depending on the driving scene. Alternatively, in consideration of the fact that the greater the relative speed with respect to the object, the greater the positional deviation on the rolling shutter camera image, the reliability R may be defined by the following equation using the relative speed v with respect to the object.

[0025]

number

[0026] Here, k2 and α2 are positive weighting coefficients that should be determined in advance based on the image and distance estimation results. The weighting coefficients k2 and α2 may be predetermined values, but may also be changed depending on the driving scene.

[0027] Furthermore, the reliability of a region in an image in which an object is reflected may be calculated from the reliability of the pair of associated pixels. The region for which the reliability is calculated may be a region in which an object is recognized, a small region obtained by dividing the image by a predetermined grid, or a region obtained by geometrically dividing a distant region and a nearby region. For example, a statistical value (e.g., an average value or a median value) of the reliability of pixels recognized as the same object in the object region on multiple images may be used as the reliability of the object region. Different images may be used for each region.

[0028] Based on the reliability calculated by the reliability calculation unit 105, the distance determination unit 106 selects the distance value to be finally adopted as the output from the distance estimation result by the first distance estimation unit 102 and the distance estimation results by the second distance estimation units 103a and 103b. For example, if the reliability value is greater than a predetermined threshold, the distance estimation result by the first distance estimation unit 102 is adopted, and if the reliability value is smaller than the threshold, the distance estimation result by the second distance estimation units 103a and 103b is adopted. The threshold value may be determined based on driving history. Furthermore, if distance estimation results are obtained by both the second distance estimation units 103a and 103b, the smaller value is adopted, and if a distance estimation result is obtained by either one of them, the value obtained is adopted.

[0029] Alternatively, in the case of a configuration further including a third distance estimation unit that measures the distance to the object using a method that does not use an image, the distance estimation result by the third distance estimation unit may be adopted if the reliability of the distance estimation using an image is low.

[0030] When the image processing device 1 is implemented in, for example, an on-vehicle electronic control device, it is a computer having an arithmetic unit, a storage device, and a communication interface. The arithmetic unit is a processor (e.g., a microcomputer) that executes programs stored in the storage device. The arithmetic unit executes predetermined programs to operate as a functional unit that provides various functions of the image processing device 1. The storage device includes a non-volatile storage area and a volatile storage area. The non-volatile storage area includes a program area that stores programs executed by the arithmetic unit, and a data area that temporarily stores data used by the arithmetic unit when executing the programs. The volatile storage area stores data used by the arithmetic unit when executing the programs. The communication interface connects to other electronic control devices via a network such as CAN or Ethernet.

[0031] Next, a processing procedure of the image processing device 1 will be described using a flowchart. Fig. 2 is a flowchart showing a distance determination process executed by the image processing device 1 shown in Fig. 1.

[0032] In step S201, the image acquisition units 101a and 101b acquire images from an on-board camera or the like.

[0033] In process S202, the imaging time calculation units 104a and 104b extract feature points, which are points characterized by changes in shading, from the image acquired in process S201. Known techniques can be used for the process of extracting feature points from the image.

[0034] In process S203, the image capturing time calculation units 104a and 104b execute a matching process to associate points that are considered to be the same points in the three-dimensional world coordinate system among the feature points extracted in process S202 between a plurality of images.

[0035] In step S204, the image capturing time calculation units 104a and 104b calculate the image capturing time of each feature point matched in step S203.

[0036] In process S205, the reliability calculation unit 105 uses the imaging time of each feature point calculated in process S204 to calculate the reliability of each feature point pair from the difference in imaging time between each feature point pair matched in process S203.

[0037] In process S206, the distance determination unit 106 determines whether the reliability calculated in process S205 is equal to or greater than a predetermined threshold value.

[0038] In step S207, if the reliability is equal to or greater than a predetermined threshold, the distance determination unit 106 outputs the distance estimation result by the first distance estimation unit 102 as a distance value.

[0039] In process S208, if the reliability is smaller than a predetermined threshold, the distance determination unit 106 outputs the distance estimation result by the second distance estimation unit 103a or the second distance estimation unit 103b as a distance value.

[0040] As described above, according to the first embodiment, a distance value to be output is selected from the estimation results of a plurality of distance estimation units based on the reliability calculated from the difference in imaging time between pairs of associated feature points, so that a distance with a small error caused by the difference in imaging time can be output.

[0041] Furthermore, distance measurement errors caused by parallax can be reduced, and accurate distances to surrounding objects can be obtained, which allows for accurate timing of danger avoidance and warnings in driving assistance systems and safe vehicle control.

[0042] Example 2 Second Embodiment A second embodiment of the present invention will be described with reference to the drawings. In the second embodiment, the same components and functions as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0043] Fig. 3 is a schematic configuration diagram of an image processing device 1 according to a second embodiment of the present invention. As shown in Fig. 3, the image processing device 1 includes image acquisition units 101a, 101b, and 101c, a first distance estimation unit 102, a second distance estimation unit 103, image capture time calculation units 104a, 104b, and 104c, reliability calculation units 105a and 105b, and a distance determination unit 106. Fig. 3 shows an example in which the image processing device 1 includes three image acquisition units 101a, 101b, and 101c, and therefore includes three image capture time calculation units and two reliability calculation units.

[0044] The first distance estimation unit 102 estimates the distance to the object using a pair of images of the object acquired by the image acquisition unit 101a and images of the object acquired by the image acquisition unit 101b. The second distance estimation unit 103 estimates the distance to the object using a pair of images of the object acquired by the image acquisition unit 101b and images of the object acquired by the image acquisition unit 101c. The first distance estimation unit 102 and the second distance estimation unit 103 may estimate the distance using images acquired by any two cameras, but may also be a combination of any two of the three cameras from which the image processing device 1 of the second embodiment acquires images. Alternatively, the distance to the object may be measured by a method that does not use images, i.e., any of the cameras may be combined with a distance measurement means that uses data acquired by a sensor other than a camera.

[0045] The image capturing time calculation units 104a to 104c calculate the image capturing times of the pixels on the images acquired by the image acquisition units 101a to 101c, respectively.

[0046] The reliability calculation unit 105a calculates the reliability using the imaging times calculated by the imaging time calculation unit 104a and the imaging times calculated by the imaging time calculation unit 104b. The reliability calculation unit 105b calculates the reliability using the imaging times calculated by the imaging time calculation unit 104b and the imaging time calculation unit 104c.

[0047] The distance determination unit 106 adopts and outputs either the distance estimation result to the object by the first distance estimation unit 102 or the distance estimation result to the object by the second distance estimation unit 103 based on the reliability calculated by the reliability calculation unit 105a and the reliability calculated by the reliability calculation unit 105b.

[0048] FIG. 4 is a flowchart of the distance determination process executed by the image processing device 1 of FIG.

[0049] In step S401, the image acquisition units 101a, 101b, and 101c acquire images from an on-board camera or the like.

[0050] In process S402, the image capturing time calculation units 104a, 104b, and 104c extract feature points, which are points characterized by changes in shading, from the image acquired in process S401. Known techniques can be used for the process of extracting feature points from the image.

[0051] In process S403, the image capturing time calculation units 104a, 104b, and 104c execute a matching process to associate points that are considered to be the same points in a three-dimensional world coordinate system among the feature points extracted in process S402 between multiple images.

[0052] In step S404, the image capturing time calculation units 104a, 104b, and 104c calculate the image capturing time of each feature point matched in step S403.

[0053] In process S405, the reliability calculation units 105a and 105b use the imaging times of each feature point calculated in process S404 to calculate the reliability of each feature point pair from the difference in imaging time between each feature point pair matched in process S403. Here, the reliability calculation unit 105a calculates the reliability from the imaging times calculated by the imaging time calculation unit 104a and the imaging times calculated by the imaging time calculation unit 104b as reliability 1, and the reliability calculation unit 105b calculates the reliability from the imaging times calculated by the imaging time calculation unit 104b and the imaging time calculation unit 104c as reliability 2.

[0054] In process S406, the distance determination unit 106 determines which of reliability 1 and reliability 2 calculated in process S405 is greater.

[0055] In process S407, if reliability 1 is equal to or greater than reliability 2, distance determination unit outputs the distance estimation result obtained by first distance estimation unit .

[0056] In process S408, if reliability 1 is smaller than reliability 2, distance determination unit 106 outputs the distance estimation result obtained by second distance estimation unit 103.

[0057] As described above, according to the second embodiment, a distance value to be output is selected from the estimation results of a plurality of distance estimation units based on the reliability calculated from the difference in imaging time between pairs of associated feature points, so that a distance with a small error caused by the difference in imaging time can be output.

[0058] In Example 2, we described the case where there are three image acquisition units, but even in the case where there are four or more image acquisition units, the reliability of pairs of pixels, regions, etc. that are matched between images is calculated from the shooting times of the pixels on the acquired images, and by adopting the distance estimated using the pair with the highest reliability, it is possible to output a distance with a small estimation error due to differences in shooting times.

[0059] The present invention is not limited to the above-described embodiments, but includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0060] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0061] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0062] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0063] 1. Image processing device 2. Reliability 101a, 101b, 101c Image acquisition section 102 First distance estimator 103, 103a, 103b, 103c Second distance estimator 104a, 104b, 104c Imaging time calculation unit 105, 105a, 105b Reliability calculation unit 106 Distance determination unit

Claims

1. an image acquisition unit that acquires a plurality of images captured by each of a plurality of cameras; a first distance estimation unit that estimates a distance to an object using the plurality of images; a second distance estimation unit that estimates a distance to the object using at least one of the plurality of images; an image capture time calculation unit that calculates image capture times of at least some of the pixels included in the plurality of images; a reliability calculation unit that calculates reliability of the plurality of images based on a difference in imaging time of the pixel in the plurality of images; an image processing device comprising: a distance determination unit that determines a distance to the object using the distance estimated by at least one of the first distance estimation unit or the second distance estimation unit and the reliability;

2. 2. The image processing device according to claim 1, the first distance estimation unit estimates a distance to the object from a parallax between images captured by two of the plurality of cameras; The image processing device is characterized in that the second distance estimation unit estimates the distance to the object using an image captured by one of the plurality of cameras.

3. 2. The image processing device according to claim 1, the image acquisition unit acquires images captured by at least three or more cameras, the first distance estimation unit estimates a distance to the object from a parallax of images captured by two of the plurality of cameras; An image processing device characterized in that the second distance estimation unit estimates the distance to the object from the parallax of images captured by two cameras that are a different combination from the camera that captured the image used by the first distance estimation unit to estimate the distance.

4. 2. The image processing device according to claim 1, further comprising a feature point extraction unit that extracts feature points from the plurality of images; The image processing device according to claim 1, wherein the image capturing time calculation unit calculates the image capturing time of the extracted feature point based on the position of the feature point on the image.

5. 5. The image processing device according to claim 4, The image processing device according to claim 1, wherein the reliability calculation unit calculates the reliability based on a difference in image capturing time of the feature points in the plurality of images.

6. 6. The image processing device according to claim 5, the reliability calculation unit calculates the reliability of at least one of the sets of images, the sets of regions in the images, and the sets of feature points based on the difference in imaging times of the feature points on the images.

7. 2. The image processing device according to claim 1, The image processing device is characterized in that the reliability calculation unit calculates the reliability of a set of areas in which the object exists in the plurality of images.

8. 2. The image processing device according to claim 1, The image processing device is characterized in that the distance determination unit selects either the distance estimation result by the first distance estimation unit or the distance estimation result by the second distance estimation unit as the distance to the object based on the reliability.

9. An image processing method executed by an image processing device, the image processing device includes a calculation device that executes predetermined processing and a storage device that can be accessed by the calculation device; The image processing method includes: an image acquisition step of acquiring a plurality of images captured by each of a plurality of cameras; a first distance estimation step of estimating a distance to an object using the plurality of images; a second distance estimation step of estimating a distance to the object using at least one of the plurality of images; an imaging time calculation step for calculating imaging times of at least some pixels included in the plurality of images; a reliability calculation step of calculating reliability of the plurality of images based on differences in imaging times of the pixels in the plurality of images; a distance determination procedure for determining the distance to the object using the distance estimated in at least one of the first distance estimation procedure or the second distance estimation procedure and the reliability.

Images