Range-finding device, moving body control system and range-finding method
The distance measuring device improves accuracy by selecting image pairs with optimal exposure conditions based on reliability estimation, addressing the issue of inaccurate distance calculations in existing systems.
Patent Information
- Application Number
- JP2024140598
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-10-26
AI Technical Summary
Existing distance measurement systems fail to determine optimal exposure conditions for parallax images, leading to inaccurate distance calculations.
A distance measuring device that selects appropriate pairs of images with different exposure conditions based on reliability estimation, including contrast, saturation, and motion shake, to calculate accurate parallax and distance.
Enables precise distance measurement by selecting suitable image pairs, reducing errors from brightness saturation and motion blur, and improving measurement accuracy.
Smart Images

Figure 2024161036000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for measuring distance using a pair of images (parallax images) having a mutual parallax. [Background technology]
[0002] A moving object such as an automobile is equipped with a distance measuring device that obtains information about distance by using the principle of stereo ranging to obtain parallax images by an imaging device. Some imaging devices can obtain HDR (High Dynamic Range) images with an expanded dynamic range by synthesizing multiple image data obtained by imaging with different exposure conditions. Patent Document 1 discloses an imaging device that can obtain parallax images by having an imaging element with a pixel configuration that can obtain signals having a phase difference between each other, and can obtain images with different exposure conditions by having high sensitivity pixels and low sensitivity pixels. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6408372 Summary of the Invention [Problem to be solved by the invention]
[0004] However, Patent Document 1 does not mention the selection of exposure conditions to be used when calculating the parallax between parallax images, which makes it difficult to determine which of the parallax images acquired with high-sensitivity pixels, the parallax image acquired with low-sensitivity pixels, and the parallax image as an HDR image is optimal for calculating the amount of parallax.
[0005] The present invention provides a distance measuring device and the like that is capable of selecting an appropriate pair of images from a plurality of pairs of images that have different exposure conditions, thereby enabling good distance measurement. [Means for solving the problem]
[0006] An information processing device according to one aspect of the present invention is characterized by having an image acquisition means for acquiring a first pair of captured images and a second pair of captured images, the first pair of captured images being captured by an imaging means and having a parallax between them, and a selection means for selecting a parallax image used to generate information relating to distance from a plurality of pairs of captured images including the first pair of captured images and the second pair of captured images. Note that a mobile body control system having the distance measuring device mounted on a mobile body and a control means for controlling the movement of the mobile body based on the information relating to distance, and further a mobile body equipped with the mobile body control system also constitute another aspect of the present invention.
[0007] According to another aspect of the present invention, there is provided an information processing method comprising the steps of acquiring a first pair of captured images and a second pair of captured images, the first pair of captured images being captured by an imaging means and having a parallax with respect to each other, and capturing the first pair of captured images with a first exposure time, and selecting a pair of captured images to be used for generating information related to distance from a plurality of pairs of captured images including the first pair of captured images and the second pair of captured images. Note that a program for causing a computer to execute processing according to the distance measuring method also constitutes another aspect of the present invention. Effect of the Invention
[0008] According to the present invention, it is possible to perform good distance measurement by selecting an appropriate pair of images from a number of pairs of images having different exposure conditions. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of a distance measuring device according to a first embodiment. [Diagram 2] 4 is a timing chart showing exposure timing of an image sensor in the first embodiment. [Diagram 3] 4 is a flowchart showing distance measurement processing in the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of an input image in the first embodiment. [Diagram 5]11 is a flowchart showing a reliability determination process in the second embodiment. [Figure 6] 13 is a flowchart showing a reliability determination process in the third embodiment. [Figure 7] 13 is a flowchart showing a reliability determination process in the fourth embodiment. [Figure 8] 13 is a flowchart showing a reliability determination process in the fifth embodiment. [Figure 9] FIG. 13 is a diagram showing the configuration of a distance measuring device and an image sensor according to a sixth embodiment. [Figure 10] FIG. 13 is a diagram showing the configuration of a moving body according to a seventh embodiment. [Figure 11] FIG. 13 is a diagram showing the configuration of a camera unit in the first and sixth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. EXAMPLES
[0011] In the first embodiment, a distance measuring device that obtains information about distance from the amount of parallax between a pair of images having a parallax between each other, which are obtained for each frame by a stereo imaging device (stereo camera), is described. The distance measuring device of this embodiment is used in digital cameras, digital video cameras, in-vehicle sensor devices, robot vision sensor devices, etc.
[0012] 1 shows the configuration of a distance measuring device 100 of this embodiment. The distance measuring device 100 includes a stereo camera (imaging means) 101, an exposure control image acquisition unit 102, a parallax reliability estimation unit (reliability acquisition means) 103, an image generation and selection unit (selection means) 104, and a stereo distance measuring unit (distance information generation means) 105. The stereo camera 101 has two camera units (imaging units) 111 and 112 arranged in parallel at a predetermined distance (baseline length) from each other. The imaging angles of view of the camera units 111 and 112 are set so as to include a common angle of view range.
[0013] 11 shows the configuration of camera units 111 and 112. Each camera unit has an imaging optical system 912, an image sensor 913, an image processing unit 914, and an image storage unit 915. The image sensor 913 is a photoelectric conversion element such as a CMOS sensor or a CCD sensor, and outputs an image signal by photoelectrically converting (capturing) the subject image formed by the imaging optical system 912. Reference numeral 920 denotes the optical axis of the imaging optical system.
[0014] The image processing unit 914 performs various image processing on the image signal read out from the imaging element 913 to generate image data (hereinafter simply referred to as image). The generated image is stored in an image storage unit 915 or transmitted to the downstream disparity reliability estimation unit 103 through a signal transmission unit (not shown). The camera unit 111 captures an image from a first viewpoint to generate a first image, and the camera unit 112 captures an image from a second viewpoint different from the first viewpoint to generate a second image. The first image and the second image form a pair of images (stereo images) having a mutual disparity.
[0015] The exposure-controlled image acquisition unit 102 has the camera units 111 and 112 and an exposure control unit (exposure control means) 113. The exposure control unit 113 simultaneously controls the exposure amount of the camera units 111 and 112. Here, the exposure control unit 113 changes the exposure time as an exposure condition for capturing images of the camera units 111 and 112. This allows each of the camera units 111 and 112 to generate an image with a first exposure time (first exposure image: hereinafter referred to as a long exposure image) and an image with a second exposure time shorter than the first exposure time (second exposure image: hereinafter referred to as a short exposure image) as images with different exposure conditions. In the following description, the two long exposure images having parallax with each other and the two short exposure images having parallax with each other generated by the camera units 111 and 112 are referred to as a pair of long exposure images (first pair of exposure images) and a pair of short exposure images (second pair of exposure images), respectively.
[0016] FIG. 2 shows the exposure time for each pixel row of the image sensor in each camera unit. The horizontal axis represents time t, and the vertical axis represents pixel row c of the image sensor. Pixel row c is a readout row that reads out image signals for each pixel row. When an image signal for the first exposure time 201 and an image signal for the second exposure time 202 are acquired in the first readout row, an image signal for the first exposure time 211 and an image signal for the second exposure time 212 are acquired in the next readout row. In this way, when an image signal for the first exposure time 2n1 and an image signal for the second exposure time 2n2 are acquired in the final readout row, the image signals of different pixel rows acquired in the first exposure times 201, 211, ..., 2n1 are combined to acquire a long-exposure image. Also, the image signals of different pixel rows acquired in the second exposure times 202, 212, ..., 2n2 are synthesized (combined) to acquire a short-exposure image.
[0017] The exposure control unit 113, the parallax reliability estimation unit 103, the image generation selection unit 104, and the stereo distance measurement unit 105 are configured by an arithmetic processing unit (a computer such as a CPU) including one or more semiconductor integrated circuits.
[0018] The pair of long-time images and the pair of short-time images acquired from the camera units 111 and 112 are transmitted to the parallax reliability estimation unit 103. The parallax reliability estimation unit 103 estimates a long-time parallax reliability, which is a reliability that is a degree of certainty of the amount of parallax calculated from the pair of long-time images by a reliability determination process described below. The parallax reliability estimation unit 103 also estimates a short-time parallax reliability, which is a reliability that is a degree of certainty of the amount of parallax calculated from the pair of short-time images by the same process. The image generation selection unit 104 receives the long-time and short-time parallax reliability and selects one of the pair of long-time images, the pair of short-time images, and the pair of HDR images (composite images) as a pair of images (hereinafter referred to as parallax calculation images) used to calculate the amount of parallax to generate information related to distance. The pair of HDR images is composed of a first HDR image generated by blending a long exposure image and a short exposure image acquired from camera unit 111, and a second HDR image generated by blending a long exposure image and a short exposure image acquired from camera unit 112.
[0019] The stereo distance measuring unit 105 calculates the amount of parallax by a known method using the pair of parallax calculation images selected by the image generation and selection unit 104, and further calculates a distance value using the amount of parallax. The distance measuring device 100 performs the above processing on image signals from all pixels of the image sensor to generate a distance image including information about the distance to a subject within the imaging angle of view. The information about the distance may be information indicating the distance itself, or may be information that can be converted into a distance using a look-up table, a conversion formula, or the like.
[0020] Specifically, the input pair of images are subjected to preprocessing including a geometric correction process, a light quantity correction process, and a noise reduction process using a bandpass filter process, and the parallax amount is calculated from the pair of images after the preprocessing, and the obtained parallax amount is converted into a distance value. In this parallax amount calculation process, a matching area having a center on the attention point coordinates is set in one of the pair of images, and a reference area having the same shape as the matching area having a center on the reference point coordinates is sequentially moved in the other image to calculate a correlation value. Then, the positional deviation amount between the attention point coordinates and the reference point coordinates with the highest correlation is set as the parallax amount. As a method for calculating the correlation value, SSD (Sum of Squared Difference), which evaluates the sum of squares of the difference between image signals, SAD (Sum of Absolute Difference), which evaluates the absolute value of the difference, and the like can be used. The set values of the attention point coordinates and the reference point coordinates are integers, but the parallax amount can be calculated with sub-pixel accuracy by peak fitting the extreme values of the correlation. In the parallax amount calculation process, the calculated parallax amount is converted into a distance value L using a geometric relational expression based on a preset base line length W between camera units 111 and 112 and a focal length f of each camera unit. In this way, a distance image is generated by calculating the parallax amount and distance value for all pixels of the imaging element.
[0021] 3 shows a distance measurement process (distance measurement method) executed by a computer of the distance measurement device (exposure control unit 113, disparity reliability estimation unit 103, image generation selection unit 104, and stereo distance measurement unit 105) according to a program. S means a step.
[0022] In S301, the disparity reliability estimation unit 103 performs a reliability determination process on the acquired input images (a pair of long-time images and a pair of short-time images). In the reliability determination process, a contrast reliability, a saturation reliability, and a motion blur reliability are calculated, which indicate the reliability as the certainty of the disparity amount calculated in the pair of images.
[0023] The contrast confidence Conf_c is confidence regarding the contrast of the subject (image) and is calculated by the following formula.
[0024]
number
[0025] 4 shows an example of an input image 401. The input image 401 is one of a pair of images, and in the figure, a matching or reference area (hereinafter, referred to as a matching / reference area) 403 centered on a focus or reference point coordinate (hereinafter, referred to as a focus / reference point coordinate) (x, y) 402 is shown. The following reliability may be calculated for at least one of the pair of images.
[0026] To calculate the contrast confidence Conf_c, first, the contrast C(x, y) of the signal brightness value I in the matching / reference area 403 centered on the attention / reference point coordinates (x, y) 402 is calculated. x and h indicate the horizontal coordinates in the input image, and y and v indicate the vertical coordinates. Iave indicates the average value of the signal brightness value I in the matching area 403.
[0027] Next, an estimated noise amount noise(x,y) in the matching / matching region 403 is calculated. q is a coefficient that is prepared in advance based on the characteristics of the image sensor, etc. Then, a contrast confidence Conf_c(x,y) of the attention point / reference point coordinates (x,y) 402 is calculated from the ratio of the contrast C(x,y) to the estimated noise amount noise(x,y). By performing these calculations for all pixels, the contrast confidence of the input image is obtained. In this manner, the contrast confidence Conf_c of each of the paired long-time image and paired short-time image is obtained.
[0028] When calculating the correlation value in calculating the amount of disparity, if the contrast is high or the amount of noise is small in the matching / reference area 403, the correlation peak becomes sharp and it is possible to calculate a highly reliable amount of disparity. Therefore, the higher the contrast confidence Conf_c is, the more the reliability of the calculated amount of disparity is determined to be high.
[0029] The saturation confidence Conf_s is a confidence regarding brightness saturation for determining whether or not there is a pixel outputting a brightness saturation or a signal brightness value equivalent thereto within the comparison / reference region 403, and is defined as follows.
[0030]
number
[0031] That is, if there is a pixel in the comparison / reference region 403 whose signal luminance value I(h,v) is equal to or greater than the signal luminance threshold Ith, low reliability is set to 0, and if there is no pixel in the comparison / reference region 403 whose signal luminance value is equal to or greater than the signal luminance threshold Ith, high reliability is set to 1. By performing this determination for all pixels, the saturation reliability of the input image can be obtained. If there is a pixel in the comparison / reference region 403 that outputs a signal luminance value that is saturated or equivalent thereto, there is a risk that the brightness-saturated region will become a pseudo texture different from the texture of the subject when the correlation value is output, and an erroneous correlation value will be calculated. In this case, a parallax amount different from the parallax amount according to the subject distance is calculated. Therefore, if there is no brightness-saturated region in the comparison / reference region 403, it is determined to be high reliability.
[0032] The motion blur reliability Conf_m is a reliability regarding motion blur for determining whether there is a pixel in the matching / reference area 403 that is significantly affected by motion blur. The amplitude of the band-pass filter processed image obtained by performing band-pass filter processing on the signal luminance value I(h,v) in the matching / reference area 403 for the attention point coordinates (x,y) is set as Conf_m(x,y). The transmission band of the band-pass filter is appropriately set to a spatial frequency band in which the influence of motion blur is to be considered. Specifically, it is set according to the moving speed of the distance measuring device, the moving speed of the main subject, and the exposure time for acquiring each second image. When the motion blur in the matching / reference area 403 is large, the texture of the subject deteriorates or the center of gravity of the subject moves during exposure, making the correlation peak dull. In other words, the calculation accuracy of the parallax amount decreases. Therefore, the smaller the amplitude of the spatial frequency band in which the influence of motion blur is to be considered, the greater the influence of motion blur, so the larger the value of the motion blur reliability Conf_m, the higher the reliability is determined to be.
[0033] The contrast reliability, saturation reliability, and motion blur reliability are integrated as shown in the following formula to determine the reliability of the disparity amount for each matching / reference region.
[0034]
number
[0035] In the above formula, α, β, and γ are weighting coefficients.
[0036] Next, in S302, the image generation selection unit 104 performs an image generation selection process to select a pair of images that are estimated to have a higher reliability of the parallax amount for each matching / reference region by comparing the integrated reliability judgment values Conf_all calculated in S301. Specifically, the image generation selection unit 104 compares the reliability judgment value Conf_all for each matching / reference region of the pair of long-time images with the reliability judgment value Conf_all for each matching / reference region of the pair of short-time images, and selects the pair of images having the matching / reference region with the higher reliability as the parallax calculation image. In other words, a parallax calculation image is selected for each matching / reference region.
[0037] If no significant difference is found as a result of comparing the reliability determination values Conf_all, the pair of HDR images described above is selected as the disparity calculation images.
[0038] Next, in S303, the stereo distance measuring unit 105 performs stereo distance measuring processing to calculate the amount of parallax for each matching / reference region using the parallax calculation image selected in S302, and further calculates a distance value to calculate a distance image.
[0039] In this way, by selecting a pair of images suitable for calculating the amount of parallax based on the parallax reliability from among a plurality of pairs of images with different exposure conditions, it becomes possible to perform distance measurement with high accuracy.
[0040] In this embodiment, it is preferable to alternately assign long and short exposure times to each pixel row of the image sensor, drive the sensor at the same read clock frequency, and generate a long exposure image and a short exposure image by synthesizing the image signals of every other pixel row after reading the image signals. This makes it possible to suppress the deviation of the exposure center of gravity of the long exposure image and the short exposure image.
[0041] In addition, the exposure conditions may be made different by providing the image sensor with first and second pixels having different exposure sensitivities, which makes it possible to simultaneously obtain a first exposure image as a high-sensitivity image corresponding to a long-exposure image and a second exposure image as a low-sensitivity image corresponding to a low-exposure image even with the same exposure time, and to avoid the influence of the difference in exposure time between the images.
[0042] Also, the contrast confidence Conf_c may be calculated using only the contrast C. Also, the motion blur confidence Conf_m may be evaluated as a frequency analysis, such as by performing a Fourier transform on the signal value of the matching / reference region for determining it and evaluating it using the amplitude of the target spatial frequency band. This allows evaluation by decomposing each frequency component, thereby further improving the distance measurement accuracy.
[0043] In addition, the distance measurement accuracy can be further improved by calculating the reliability of not only one of the pair of images but also the reliability of the other image, and selecting the paired image with the highest overall reliability rating (average value, maximum value, etc.) as the disparity calculation image.
[0044] Furthermore, when generating an HDR image by blending a long-time image and a short-time image, the blending ratio may be determined using the ratio of their contrast reliability. Since the contrast reliability affects the calculation accuracy of the parallax amount under the same conditions for both the long-time image and the short-time image, it is preferable to determine the blending ratio as described above from the viewpoint of improving distance measurement accuracy. For example, when the contrast reliability of the long-time image / the contrast reliability of the short-time image=k, HDR image = k / (1+k) × long-exposure image + 1 / (1+k) × short-exposure image, An HDR image may be generated as follows.
[0045] In addition, in selecting the parallax calculation image, a pair of images may be selected for each region for which correlation calculation is performed to calculate the amount of parallax. This allows an appropriate amount of parallax to be calculated according to the type and arrangement of the subject, etc., of the scene to be captured, and improves the accuracy of distance measurement.
[0046] Furthermore, the parallax amount for each of the first and second pairs of exposure images under different exposure conditions may be calculated prior to the stereo distance measurement process, and then a pair of images for distance measurement may be selected based on the reliability, and distance measurement may be performed using the parallax amount for the selected pair of images. When this process is performed using a logic circuit such as an FPGA (Field Programmable Gate Array), it is preferable from the viewpoint of implementation cost to calculate the parallax amount for the entire pair of images and select the parallax amount to be used for distance measurement later, rather than switching the pair of images for each coordinate (x, y) and calculating the parallax amount. EXAMPLES
[0047] In Example 2, the parallax calculation image is selected using the saturation reliability, motion blur reliability, and contrast reliability described in Example 1 in order. The configuration of the distance measurement device is the same as that of the distance measurement device 100 shown in FIG. 1, and the same reference numerals are given to the components common to Example 1. The distance measurement process is the same as that shown in the flowchart of FIG. 3, except for S301.
[0048] The flowchart of FIG. 5 shows the reliability determination process executed by the parallax reliability estimation unit 103 in this embodiment instead of S301 in FIG. 3.
[0049] In S501, the parallax reliability estimation unit 103 calculates the saturation reliability Conf_s for the pair of long-exposure images and the pair of short-exposure images as input images. Then, it is determined whether there is a collation / reference area where Conf_s = 0 in each of the long-exposure image and the short-exposure image, that is, whether there is a luminance saturation area. If the luminance saturation area exists (True), the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. This is because the short-exposure image is less affected by luminance saturation than the long-exposure image and is therefore suitable as the parallax calculation image. On the other hand, if the luminance saturation area does not exist (FALSE), the parallax reliability estimation unit 103 proceeds to S502.
[0050] In S502, the parallax reliability estimation unit 103 calculates the motion blur reliability Conf_m for the pair of long-exposure images and the pair of short-exposure images. Then, using the motion blur determination threshold th_m, it is determined whether there is a collation / reference area where Conf_m < th_m, that is, whether there is a collation / reference area where the influence of motion blur is large and the value of Conf_m is small. If a collation / reference area with a large influence of motion blur (Conf_m < th_m) exists (TRUE), the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. This is because the short-exposure image is less affected by motion blur than the long-exposure image and is therefore suitable as the parallax calculation image. On the other hand, if a collation / reference area with a large influence of motion blur does not exist (FALSE), the parallax reliability estimation unit 103 proceeds to S503.
[0051] In S503, the parallax reliability estimation unit 103 calculates the contrast reliability Conf_c for the pair of long-exposure images and the pair of short-exposure images, and compares the contrast reliability Conf_c_L of the short-exposure images with the contrast reliability Conf_c_H of the long-exposure images. Further, the difference between these contrast reliabilities is compared with the contrast determination threshold th_c(0) as a predetermined value.
[0052] When Conf_c_L - Conf_c_H ≥ th_c, that is, when the contrast reliability of the short-exposure images is significantly higher than that of the long-exposure images, the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. Also, when Conf_c_H - Conf_c_L ≥ th_c, that is, when the contrast reliability of the long-exposure images is significantly higher than that of the short-exposure images, the parallax reliability estimation unit 103 proceeds to S512 and outputs a determination result of selecting the pair of long-exposure images as the parallax calculation image. Further, when |Conf_c_L - Conf_c_H| < th_c, that is, when there is no significant difference in the contrast reliabilities between the long-exposure images and the short-exposure images, the parallax reliability estimation unit 103 proceeds to S513 and outputs a determination result of selecting the pair of HDR images as the parallax calculation image.
[0053] The determination results of S511, S512, and S513 are passed to the image generation selection unit 104 (S302 in FIG. 1). The image generation selection unit 104 selects one of the pair of short-exposure images, the pair of long-exposure images, and the pair of HDR images as the parallax calculation image based on the determination result. The stereo distance measurement unit 105 (S303 in FIG. 1) performs stereo distance measurement processing using the selected pair of images.
[0054] In this way, by determining multiple types of reliability in order and selecting a parallax calculation image, it is possible to speed up the process compared to the case where all the reliability is calculated and then a parallax calculation image is selected. The contrast affects the accuracy of parallax calculation (i.e., distance measurement) under similar conditions regardless of whether it is a short-time image or a long-time image, but the short-time image is less affected by brightness saturation and motion blur than the long-time image. For this reason, as in this embodiment, it is preferable to select a parallax calculation image based on the saturation reliability and motion blur reliability before selecting a parallax calculation image based on the contrast reliability. EXAMPLES
[0055] In the third embodiment, a disparity calculation image is selected using only the contrast reliability described in the first embodiment. The configuration of the distance measuring device is the same as that of the distance measuring device 100 shown in Fig. 1, and the same reference numerals are used for the components common to the first embodiment. The distance measuring process is the same as that shown in the flowchart of Fig. 3, except for S301.
[0056] The flowchart in FIG. 6 shows a reliability determination process executed by the disparity reliability estimation unit 103 in this embodiment instead of S301 in FIG.
[0057] 5, the disparity reliability estimation unit 103 calculates contrast reliability Conf_c for a pair of long time duration images and a pair of short time duration images as input images, and compares contrast reliability Conf_c_L and Conf_c_H of the short time duration images and the long time duration images. Also, the difference between the contrast reliability is compared with the contrast determination threshold th_c(0).
[0058] Then, when Conf_c_L - Conf_c_H ≥ th_c, the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting a pair of short-exposure images as the parallax calculation image. When Conf_c_H - Conf_c_L ≥ th_c, it proceeds to S512 and outputs a determination result of selecting a pair of long-exposure images as the parallax calculation image. Further, when |Conf_c_L - Conf_c_H| < th_c, it proceeds to S513 and outputs a determination result of selecting a pair of HDR images as the parallax calculation image.
[0059] By selecting the parallax calculation image using only the contrast reliability in this way, the computational amount can be significantly reduced compared to those in the first and second embodiments. In addition, the probabilities of occurrence of the effects of luminance saturation and motion blur depend on the imaging scene such as the imaging location, time, and subject. On the other hand, the contrast always affects the subject and ambient light under the same conditions in both the short-exposure image and the long-exposure image. From these facts, the selection of the parallax calculation image may be performed only based on the contrast reliability.
Embodiment
[0060] In the fourth embodiment, the parallax calculation image is selected using the saturation reliability and the contrast reliability described in the first embodiment. The configuration of the distance measurement device is the same as the distance measurement device 100 shown in FIG. 1, and the same reference numerals are given to the components common to the first embodiment. The distance measurement process is the same as that shown in the flowchart of FIG. 3 except for S301.
[0061] The flowchart of FIG. 7 shows the reliability determination process executed by the parallax reliability estimation unit 103 in this embodiment instead of S301 in FIG. 3.
[0062] In S701, the parallax reliability estimation unit 103 calculates the saturation reliability Conf_s for the pair of long-exposure images and the pair of short-exposure images as the input images in the same way as S501 in FIG. 7, and determines whether there is a collation / reference area with Conf_s = 0 in each of the long-exposure image and the short-exposure image. If the luminance saturation area exists and the result is True, the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. If the luminance saturation area does not exist and the result is FALSE, it proceeds to S702.
[0063] In S702, the parallax reliability estimation unit 103 calculates the contrast reliability Conf_c for the pair of long-exposure images and the pair of short-exposure images as the input images in the same way as S503 in FIG. 5, and compares the contrast reliabilities Conf_c_L and Conf_c_H of the short-exposure image and the long-exposure image. Also, the difference between these contrast reliabilities is compared with the contrast determination threshold th_c(0).
[0064] If Conf_c_L - Conf_c_H ≧ th_c, the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. If Conf_c_H - Conf_c_L ≧ th_c, it proceeds to S512 and outputs a determination result of selecting the pair of long-exposure images as the parallax calculation image. Furthermore, if |Conf_c_L - Conf_c_H| < th_c, it proceeds to S513 and outputs a determination result of selecting the pair of HDR images as the parallax calculation image.
[0065] If there is a luminance saturation area in the collation / reference area for calculating the contrast reliability, the contrast of the subject cannot be correctly calculated, leading to misjudgment. Therefore, this embodiment can prevent misjudgment of the contrast reliability due to luminance saturation and obtain an effect of reducing the calculation amount.
Embodiment
[0066] In Example 5, a parallax calculation image is selected using saturation reliability and contrast reliability. After that, when it cannot be selected based on contrast reliability, a parallax calculation image is selected based on motion blur reliability. The configuration of the distance measurement device is the same as that of the distance measurement device 100 shown in FIG. 1, and the same reference numerals are given to the components common to Example 1. The distance measurement process is the same as that shown in the flowchart of FIG. 3, except for S301.
[0067] The flowchart of FIG. 8 shows the reliability determination process executed by the parallax reliability estimation unit 103 in this embodiment instead of S301 in FIG. 3.
[0068] In S801, the parallax reliability estimation unit 103 calculates the saturation reliability Conf_s for the pair of long-exposure images and the pair of short-exposure images in the same manner as in S501 of FIG. 7, and determines whether there is a collation / reference region with Conf_s = 0 in each of the long-exposure image and the short-exposure image. If the luminance saturation region exists (True), the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. If the luminance saturation region does not exist (FALSE), it proceeds to S802.
[0069] In S802, the parallax reliability estimation unit 103 calculates the contrast reliability Conf_c for the pair of long-exposure images and the pair of short-exposure images as the input images in the same manner as in S503 of FIG. 5, and compares the contrast reliabilities Conf_c_L and Conf_c_H of the short-exposure image and the long-exposure image. Also, the difference between these contrast reliabilities is compared with the contrast determination threshold th_c(0). When Conf_c_L - Conf_c_H ≧ th_c, the parallax reliability estimation unit 103 proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. When Conf_c_H - Conf_c_L ≧ th_c, it proceeds to S512 and outputs a determination result of selecting the pair of long-exposure images as the parallax calculation image. Furthermore, when |Conf_c_L - Conf_c_H| < th_c, it proceeds to S803.
[0070] In S803, similar to S502 in FIG. 5, the parallax reliability estimation unit 103 calculates the motion blur reliability Conf_m for the pair of long-exposure images and the pair of short-exposure images. Then, using the motion blur determination threshold th_m, it determines whether there is a matching / reference region where Conf_m < th_m. If TRUE, i.e., there is a matching / reference region with a large influence of motion blur (Conf_m < th_m), it proceeds to S804. If FALSE, i.e., there is no matching / reference region with a large influence of motion blur, it proceeds to S513 and outputs a determination result of selecting the pair of HDR images as the parallax calculation image.
[0071] In S804, the parallax reliability estimation unit 103 compares the contrast reliabilities Conf_c_L and Conf_c_H of the short-exposure image and the long-exposure image again. If the contrast reliability Conf_c_L of the short-exposure image is higher, it proceeds to S511 and outputs a determination result of selecting the pair of short-exposure images as the parallax calculation image. If the contrast reliability Conf_c_H of the long-exposure image is higher, it proceeds to S512 and outputs a determination result of selecting the pair of long-exposure images as the parallax calculation image.
[0072] When synthesizing the long-exposure image and the short-exposure image that include an image with a large influence of motion blur, artifacts occur around the subject with a large influence of motion blur, leading to a decrease in the calculation accuracy of the parallax amount. Therefore, according to this embodiment, based on the selection of the parallax calculation image by the luminance saturation region and the contrast reliability, when selection cannot be made from these, the occurrence of artifacts due to image synthesis is prevented, and a pair of HDR images suitable for the calculation of the parallax amount is selected.
Example
[0073] In the first embodiment, the stereo camera is configured using two camera units 111 and 112, but in the present embodiment, the stereo camera is configured using a single camera unit. Fig. 10 also shows the configuration of a camera unit 911 in this embodiment. However, in this embodiment, the configuration of an image sensor 913 is different from that in the first embodiment, and has a configuration capable of performing image plane phase difference ranging using a pupil division method. In Fig. 10, the z direction indicates the optical axis direction along which an optical axis 920 extends, and the x direction and y direction indicate two directions that are perpendicular to the optical axis 920 and perpendicular to each other.
[0074] 9(b) shows an xy cross section of the image sensor 913. The image sensor 913 is configured by arranging a plurality of unit pixels 930 in the x and y directions. The light receiving layer of the unit pixel 930 is configured by a first photoelectric conversion unit 931 and a second photoelectric conversion unit 932 as two photoelectric conversion units.
[0075] 9(c) shows a II' cross section of one unit pixel 930. The unit pixel 930 has a light guide layer 941 and a light receiving layer 942. The light guide layer 941 is provided with a microlens 943 for efficiently guiding a light flux incident on the unit pixel 930 to the photoelectric conversion units 931 and 932. Although not shown, the light guide layer 941 also has a color filter that transmits light of a predetermined wavelength band, wiring for driving pixels and reading signals, and the like arranged therein. The light receiving layer 942 has first and second photoelectric conversion units 931 and 932 arranged therein for photoelectrically converting light incident through the light guide layer 941.
[0076] In a camera unit 911 equipped with a single image sensor 913 having such a unit pixel structure and a single imaging optical system 912, a first image and a second image having different viewpoints can be obtained by a pupil division method.
[0077] FIG. 9(d) shows the principle of the pupil division method. In this figure, an exit pupil 950 of an imaging optical system 912 and a unit pixel 930 in the vicinity of a representative central image height among a plurality of unit pixels 930 in an image sensor 913 are shown. A microlens 943 in the unit pixel 930 is arranged so that the exit pupil 950 and a light receiving layer 942 are optically conjugate with each other. As a result, a light beam passing through a first pupil region 951, which is a partial pupil region included in the exit pupil 950, enters a first photoelectric conversion unit 931, and a light beam passing through a second pupil region 952, which is another partial pupil region, enters a second photoelectric conversion unit 932. Even if the unit pixel 930 is located at a peripheral image height position, the principal ray enters the unit pixel 930 at an angle, but the corresponding relationship between the partial pupil region, the light beam, and the photoelectric conversion unit does not change.
[0078] A plurality of unit pixels 930 are arranged in the image sensor 913, and a first image is generated by reading out and combining signals generated by photoelectric conversion in the first photoelectric conversion unit 931 of each unit pixel. Similarly, a second image is generated by reading out and combining signals generated by photoelectric conversion in the second photoelectric conversion unit 932 of each unit pixel.
[0079] The distance measuring device 100' of this embodiment includes an exposure control image acquisition unit 102' including the above-mentioned camera unit 911 and exposure control unit 113', a disparity reliability estimation unit 103, an image generation selection unit 104, and a stereo distance measuring unit 105 also shown in Figure 1.
[0080] By the exposure control unit 113' controlling the exposure amount of the image sensor 913 in the manner described in the first embodiment, it is possible to obtain a pair of long exposure images as the first and second exposure images, and a pair of short exposure images as the first and second exposure images.
[0081] In the distance measuring device 100' of this embodiment, the amount of parallax can be calculated and a distance image can be generated by performing the processing described in any of Examples 1 to 5 using a pair of long exposure images and a pair of short exposure images acquired from a single camera unit 911.
[0082] By using a single camera unit as in this embodiment, the influence of geometric deviation other than parallax between paired images is smaller than when using two camera units. Therefore, in this embodiment, by appropriately selecting a pair of images suitable for calculating the amount of parallax from multiple pairs of images with different exposure conditions, it is possible to perform distance measurement with higher accuracy. EXAMPLES
[0083] Fig. 10(a) shows a vehicle 1000 which is a moving body, and Fig. 10(b) shows a control system mounted on the vehicle 1000. Note that the moving body is not limited to an automobile, and may be a train, an airplane, a ship, a small mobility vehicle, or a robot such as an AGV (Automatic Guided Vehicle).
[0084] 10(a), a vehicle 1000 includes a distance measuring device 1010 (100, 100') described in the first to sixth embodiments, a millimeter wave radar device 1020, a LiDAR (Light Detection and Ranging) device 1030, and a vehicle information measuring instrument 1040. The vehicle 1000 also includes a route generation ECU (Electronic Control Unit) 1050 and a vehicle control ECU 1060. The route generation ECU 1050 and the vehicle control ECU 1060 may be configured by a CPU.
[0085] The distance measuring device 1010 captures images of the surrounding environment including the road on which the vehicle 1000 is traveling, and generates image information including a pair of images having parallax between them (a long-exposure image, a short-exposure image, and an HDR image). It also generates distance image information (hereinafter referred to as distance information) including information on the distance to a subject for each pixel of the imaging element. The distance measuring device 1010 outputs the image information and distance information to the route generation ECU 1050. The distance measuring device 1010 is disposed near the top edge of the windshield of the vehicle 1000 as shown in the figure, and captures an image of an area in a predetermined angular range (imaging angle of view) toward the front of the vehicle 1000.
[0086] As described in the first embodiment, the information on the distance may be information indicating the distance itself, or may be information that can be converted into a distance. Information that assigns the distance to a predetermined integer value may be output to the route generation ECU 1050. Furthermore, information on an optically conjugate distance value (defocus amount from the image sensor to the conjugate point) that can be converted into a distance to a subject, or information on the distance from the optical system to the conjugate point (distance from the image side principal point to the conjugate point) may be output to the route generation ECU 1050. The vehicle information measuring instrument 1040 measures the traveling speed and detects the steering angle of the vehicle 1000, and outputs measurement information indicating these to the route generation ECU 1050.
[0087] The route generation ECU 1050 is configured using a logic circuit. The route generation ECU 1050 acquires measurement information from the vehicle information measuring instrument 1040, image information and distance information from the distance measuring device 1010, distance information from the radar device 1020, and distance information from the LiDAR device 1030. Based on the acquired information, the route generation ECU 1050 generates target route information related to at least one of a target travel trajectory and a target travel speed of the vehicle 1000, and outputs the information to the vehicle control ECU 1060 in sequence.
[0088] The vehicle control ECU 1060 performs driving control to control the rotation speed of the engine or motor and the rotation angle of the steering wheel of the vehicle 1000 so that the running speed and steering angle of the vehicle 1000 become control target values based on the target route information. The distance measuring device 1010 and the vehicle control ECU 1060 configure an in-vehicle system as a mobile object control system.
[0089] In addition, the vehicle 1000 may be equipped with an HMI (Human Machine Interface) 1070 that displays an image or notifies the driver 1001 by voice. In this case, the target route information generated by the route generation ECU 1050 can be displayed or notified to the driver 1001 through the HMI 1070.
[0090] The distance measuring device 1010 selects a pair of images suitable for calculating the amount of parallax from multiple pairs of images having different exposure conditions and outputs highly accurate distance information to the route generation ECU 1050, thereby improving the accuracy of the target route information generated by the route generation ECU 1050 and achieving safer vehicle driving control.
[0091] The distance measuring device 1010 mounted on a moving body such as a vehicle captures scenes where brightness saturation is likely to occur, such as the exit of a tunnel or the headlights of an oncoming vehicle at night, or objects with significant motion blur such as an oncoming vehicle. Even when capturing images of such scenes or objects, a highly accurate distance measuring process can be performed with reduced effects of brightness saturation and motion blur by selecting a pair of images suitable for calculating the amount of parallax for each angle of view (matching / reference area) of the pair of images as described in the first to sixth embodiments.
[0092] The vehicle control ECU 1060 may use the distance information obtained by the distance measuring device 1010 to determine whether the distance to the object is within a predetermined distance range. The vehicle control ECU 1060 determines that there is a "possibility of collision" when the object is present within the predetermined distance range, and determines that there is no possibility of collision when there is no object within the predetermined distance. If it determines that there is a "possibility of collision," the vehicle control ECU 1060 may control the brakes or steering of the vehicle 1000 based on the determination result, or may issue a warning to the driver 1001. The warning may include displaying warning information on a display screen or applying vibration to a seat belt or steering.
[0093] Also, a notification unit (notification means) may be provided for notifying a notification destination such as a manufacturer or a dealer of the moving body when the vehicle 1000 collides with an object. As the notification unit, for example, a unit that transmits information related to the collision between the vehicle 1000 and an object to a preset external notification destination by e-mail or the like may be adopted.
[0094] The above embodiment includes the following configurations.
[0095] (Configuration 1) an imaging means for capturing images to generate a pair of images having a parallax between them; a distance information generating means for calculating a parallax amount from the pair of images and generating information regarding distance from the parallax amount; an exposure control means for causing the imaging means to generate, as the pair of images, a first pair of exposure images and a second pair of exposure images having different exposure conditions; a reliability acquisition means for acquiring reliability of a parallax amount calculated from each of the first pair of exposed images and the second pair of exposed images; a selection unit that selects, based on the reliability, a disparity calculation image as the pair of images from which a disparity amount for generating information regarding the distance is obtained. (Configuration 2) The distance measuring device according to configuration 1, characterized in that the selection means selects, as the parallax calculation image, one of the first pair of exposure images, the second pair of exposure images, or a pair of composite images obtained by combining the first pair and the second pair of exposure images. (Configuration 3) 3. The distance measuring device according to configuration 1 or 2, wherein the reliability is at least one of a reliability regarding contrast, a reliability regarding brightness saturation, and a reliability regarding motion blur. (Configuration 4) 4. The distance measuring device according to configuration 3, wherein the selection means selects the parallax calculation image based on a reliability obtained by integrating the reliability regarding the contrast, the reliability regarding the brightness saturation, and the reliability regarding the motion blur. (Configuration 5) 4. The distance measuring device according to configuration 3, wherein the selection means selects the parallax calculation image using the reliability regarding the luminance saturation, the reliability regarding the motion blur, and the reliability regarding the contrast in this order. (Configuration 6) 4. The distance measuring device according to configuration 3, wherein the selection means selects the parallax calculation image using the reliability regarding the luminance saturation and the reliability regarding the contrast in this order. (Configuration 7) The distance measuring device described in configuration 3, characterized in that when a difference in reliability regarding the contrast between the first pair of exposure images and the second pair of exposure images is smaller than a predetermined value, the selection means selects, as the disparity calculation image, a pair of composite images obtained by combining the first pair and the second pair of exposure images. (Configuration 8) 2. The distance measuring device according to configuration 1, wherein the exposure control means causes the imaging means to generate the first and second pairs of exposure images by changing an exposure time of an imaging element in the imaging means. (Configuration 9) The distance measuring device described in configuration 1, characterized in that the exposure control means causes the imaging means to generate the first pair and the second pair of exposure images by exposing a first pixel and a second pixel having different sensitivities provided in an imaging element in the imaging means for the same exposure time. (Configuration 10) 2. The distance measuring device according to configuration 1, wherein the imaging means is composed of two imaging units arranged at a base length apart from each other, each having an image forming optical system and an imaging element. (Configuration 11) 2. The distance measuring device according to configuration 1, wherein the imaging means comprises a single imaging optical system and a single imaging element having a pair of photoelectric conversion units for each pixel. (Configuration 12) A distance measuring device according to configuration 1 mounted on a moving object; A mobile object control system comprising a control means for controlling the movement of the mobile object based on the information regarding the distance. (Configuration 13) The mobile object control system is characterized in that the control means determines a possibility of a collision between the mobile object and an object based on the information regarding the distance. (Configuration 14) 14. The mobile body control system according to claim 13, wherein the control means controls braking or steering of the mobile body when it determines that there is a possibility of a collision between the mobile body and the object. (Configuration 15) 14. The mobile body control system according to claim 13, wherein the control means issues a warning to a driver of the mobile body when it determines that there is a possibility of a collision between the mobile body and the object. (Configuration 16) A mobile unit control system comprising: a notification means for notifying an outside party of information regarding a collision between the mobile unit and the object. (Configuration 17) A vehicle control system according to any one of configurations 12 to 16, A moving body capable of moving while holding the distance measuring device.
[0096] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0097] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]
[0098] 100,100′,1010 Rangefinder 111,112,911 Camera unit 113, 113' Exposure control unit 103 Disparity Reliability Estimation Unit 104 Image Generation Selection Unit 105 Stereo distance measurement unit
Claims
1. an image acquisition means for acquiring a pair of captured images having a parallax between them, the pair of captured images being captured by an image capture means, the first pair of captured images being the pair of captured images captured with a first exposure time, and a second pair of captured images being the pair of captured images captured with a second exposure time; and a selection means for selecting the pair of captured images to be used for generating information regarding distance from a plurality of pairs of captured images including the first pair of captured images and the second pair of captured images.
2. 2. The information processing apparatus according to claim 1, further comprising a generating unit that calculates an amount of parallax based on the pair of captured images and generates information regarding the distance based on the amount of parallax.
3. 3. The information processing apparatus according to claim 2, further comprising a control means for controlling an exposure condition of said image pickup means.
4. The information processing device according to claim 1, characterized in that the selection means selects, as the pair of captured images used to generate information regarding the distance, one of the first pair of captured images, the second pair of captured images, or a composite image obtained by combining the first pair of captured images and the second pair of captured images.
5. a reliability acquisition means for acquiring reliability, which is a degree of certainty of a parallax amount, for the first pair of captured images and the second pair of captured images; 2 . The information processing apparatus according to claim 1 , wherein the reliability is at least one of a reliability regarding contrast, a reliability regarding brightness saturation, and a reliability regarding motion blur.
6. 6. The information processing device according to claim 5, wherein the selection means selects the pair of captured images used to generate information related to the distance based on a reliability obtained by integrating the reliability related to the contrast, the reliability related to the brightness saturation, and the reliability related to the motion blur.
7. 6. The information processing device according to claim 5, wherein the selection means selects the pair of captured images used to generate information related to the distance by using the reliability regarding the brightness saturation, the reliability regarding the motion blur, and the reliability regarding the contrast in this order.
8. 6. The information processing apparatus according to claim 5, wherein the selection means selects the pair of captured images used to generate information related to the distance using the reliability regarding the brightness saturation and the reliability regarding the contrast in this order.
9. The information processing device according to claim 5, characterized in that, when a difference in reliability regarding the contrast between the first pair of captured images and the second pair of captured images is smaller than a predetermined value, the selection means selects a composite image obtained by combining the first pair of captured images and the second pair of captured images as the pair of captured images to be used for generating information regarding the distance.
10. 2 . The information processing apparatus according to claim 1 , wherein the image acquisition means acquires the first pair of captured images and the second pair of captured images by changing an exposure time of an image sensor in the imaging means.
11. The information processing device according to claim 1, characterized in that the image acquisition means acquires the first pair of captured images and the second pair of captured images by exposing a first pixel and a second pixel having different exposure sensitivities provided in an imaging element of the imaging means for the same exposure time.
12. 2. The information processing apparatus according to claim 1, wherein said image pickup means is composed of two image pickup units arranged at a base length from each other, each having an image forming optical system and an image pickup element.
13. 2. The information processing apparatus according to claim 1, wherein the image capturing means comprises a single imaging optical system and a single image sensor having a pair of photoelectric conversion units for each pixel.
14. The information processing device according to claim 1, which is mounted on a moving body; A mobile object control system comprising a control means for controlling the movement of the mobile object based on the information regarding the distance.
15. 15. The mobile body control system according to claim 14, wherein the control means determines the possibility of a collision between the mobile body and an object based on the information regarding the distance.
16. 16. The mobile body control system according to claim 15, wherein the control means controls braking or steering of the mobile body when it is determined that there is a possibility of a collision between the mobile body and the object.
17. 16. The mobile body control system according to claim 15, wherein the control means issues a warning to a driver of the mobile body when it is determined that there is a possibility of a collision between the mobile body and the object.
18. 16. The mobile unit control system according to claim 15, further comprising a notification means for notifying an outside party of information regarding a collision between said mobile unit and said object.
19. A vehicle control system according to any one of claims 14 to 18, A mobile body capable of moving while holding the information processing device.
20. A pair of captured images having a parallax between them captured by an imaging means, acquiring a first pair of captured images captured with a first exposure time and a second pair of captured images captured with a second exposure time; and selecting, from a plurality of pairs of captured images including the first pair of captured images and the second pair of captured images, a pair of captured images to be used for generating information regarding distance.
21. 21. A program for causing a computer to execute a process according to the information processing method according to claim 20.
22. an image acquisition means for acquiring a pair of captured images having a parallax between them, the pair of captured images being captured by an image capture means, the first pair of captured images being the pair of captured images captured with a first exposure time, and a second pair of captured images being the pair of captured images captured with a second exposure time; a reliability obtaining means for obtaining a reliability, which is a degree of certainty of a parallax amount, for each of the first pair of captured images and the second pair of captured images; and a selection means for selecting one of the first pair of captured images and the second pair of captured images for use in generating information regarding distance based on the reliability.
23. an image acquisition means for acquiring a pair of captured images having a parallax between them, the pair of captured images being captured by an image capture means, the first pair of captured images being the pair of captured images captured with a first exposure time, and a second pair of captured images being the pair of captured images captured with a second exposure time; a reliability obtaining means for obtaining a reliability, which is a degree of certainty of a parallax amount, for each of the first pair of captured images and the second pair of captured images; and a selection means for selecting, based on the reliability, one of the first pair of captured images, the second pair of captured images, or a composite image obtained by combining the first pair of captured images and the second pair of captured images, for use in generating information regarding distance.
Citation Information
Patent Citations
Image processing device, image processing method and program
JP2015073185A
Imaging apparatus and image processing method
JP2015133668A
Distance range calculation apparatus, distance reliability calculation device, distance image signal correction apparatus, distance image signal quantification apparatus, imaging apparatus and method
JP2016173322A
Car outside environment recognition device
JP2019168821A
Three-dimensional imaging apparatus and three-dimensional imaging condition adjustment method
JP2020139767A
