Image processing device and image processing method

Abstract

To provide an image processing device that correctly calculates parallax by excluding the influence of diffused light components even when a difference in the diffused light components occurs due to different lens characteristics for the left and right sides of a stereo camera that photographs a light source.SOLUTION: An image processing device of the present invention includes: a light source area identification unit that identifies the area where a light source is photographed in each of multiple images obtained simultaneously by multiple cameras with overlapping fields of view; and a parallax calculation unit that weights pixels in the light source area according to their brightness values and calculates parallax by performing parallax matching of the light source area using each weighted pixel.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method for calculating the three-dimensional position of an object based on a pair of images captured synchronously by a stereo camera. [Background technology]

[0002] In recent years, automobiles equipped with advanced driver assistance systems and autonomous driving systems are becoming increasingly common. These systems estimate the three-dimensional positions of captured objects around the vehicle, such as other vehicles, pedestrians, obstacles, road centerlines, lane boundaries, etc., and control the vehicle's driving, braking, steering, etc. based on these estimation results.

[0003] A stereo camera is known as one type of external recognition device used in these systems. In a stereo camera, multiple cameras placed at different positions capture images of the same object from different viewpoints synchronously, and the three-dimensional position of the captured object is estimated by calculating the distance to the object based on the difference in how the object appears in the multiple captured images, known as parallax.

[0004] Methods such as block matching are used to calculate disparity. This method uses one of a pair of images captured synchronously by a pair of cameras as a reference image, extracts a specified small area on the reference image, and searches for the same small area in the other reference image. To determine whether the small areas are identical, a cost function such as SAD (Sum of Absolute Difference) is used. However, the use of these disparity calculation methods requires that the optical characteristics of each camera are identical.

[0005] While search during block matching is typically performed in pixel units, Patent Document 1 describes a technology for improving ranging resolution through post-stage processing. The abstract of this document states that the problem is to "eliminate the degradation of ranging resolution at long distances caused by pixel-based parallax obtained by processing stereo images, thereby improving ranging resolution from short to long distances." It also states that the solution is to "perform stereo matching on a pair of images captured by a stereo camera 10, in which a stereo processing unit 30 calculates the city-block distance for each small region of each image and determines the correlation between them to identify corresponding small regions. A distance image is generated in which pixel shifts (parallax) occurring depending on the distance to the target are visualized as distance data. Furthermore, a recognition processing unit 40 performs stereo matching using data from the reference image and the comparison image to determine parallax of less than one pixel (sub-pixel components), and interpolates the pixel-based parallax obtained from the distance image with a resolution of less than one pixel. This eliminates the degradation of ranging resolution at long distances and ensures ranging accuracy from short to long distances." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-283753 Summary of the Invention [Problem to be solved by the invention]

[0007] To ensure distance measurement accuracy using the technology of Patent Document 1, it is necessary to assume that the optical characteristics of the left and right cameras of the stereo camera are identical. However, in reality, the optical characteristics of each camera are not necessarily identical, so if the parallax is calculated under the assumption that the optical characteristics of each camera are identical, the result of the parallax calculation will contain errors.

[0008] One of the reasons why the optical characteristics of each camera differ is the difference in the diffusion characteristic F of each camera's lens. As shown in Figure 1, in camera 2, which has lens 21 (simply shown as a single lens, but may be a combination of multiple lenses) and image sensor 22 (e.g., a CMOS image sensor), light emitted from an ideal point light source S is projected onto image sensor 22 through lens 21, and image P is generated.

[0009] In this case, since a point light source S that has no size is being imaged, ideally it is expected that the light source will be imaged only in one pixel where the point light source S is imaged on the image sensor 22, as in the image Pi of the "ideal model." However, in reality, the light from the point light source S is diffused around the imaging position due to the diffusion characteristic F of the lens 21, so that the light resulting from the point light source S is imaged over a wide range, with the brightness decreasing as it moves toward the periphery, as in the image Pr of the "actual model."

[0010] Therefore, if the diffusion characteristics F of the left and right cameras of a stereo camera are different, even if a point light source S of the same brightness is captured from the same position, each camera will capture a different image. As a result, when calculating parallax using stereo cameras with different diffusion characteristics F, block matching is performed based on the different diffusion components for each image, so the parallax of the light source cannot be calculated accurately and the three-dimensional position of the light source cannot be accurately estimated. In particular, when the light source is far away, due to the characteristics of stereo cameras, even a small parallax error can lead to a large ranging error. Therefore, in processes that measure the distance to a distant object and use the distance for subsequent control, such as a vehicle-following function using an in-vehicle stereo camera, parallax errors caused by differences in the diffusion characteristics F of each camera become a major problem.

[0011] One possible solution to this problem would be to pair lenses so that the left and right cameras of the stereo camera have the same diffusion characteristic F; however, because there is variation in the diffusion characteristic F even among lenses of the same design value and from the same lot, pairing lenses with matching diffusion characteristics requires a great deal of effort, as it is necessary to measure the diffusion characteristic F for each lens and then find and pair lenses with similar characteristics.

[0012] The present invention has been made in consideration of such problems, and aims to provide an image processing device and an image processing method that can correctly calculate disparity even if the diffusion characteristics of each camera that makes up a stereo camera differ. [Means for solving the problem]

[0013] In order to achieve the above object, the present invention comprises a light source area identification unit that identifies an area in which a light source is captured in each of multiple images obtained simultaneously by multiple cameras with overlapping fields of view, and a disparity calculation unit that weights pixels in the light source area according to the luminance value of the pixels, and determines the disparity by performing disparity matching of the light source area using each weighted pixel. [Effects of the Invention]

[0014] According to the image processing device and image processing method of the present invention, it is possible to accurately calculate parallax even if the diffusion characteristics of the cameras constituting the stereo camera are different. Problems, configurations and effects other than those described above will become clearer in the following examples. [Brief explanation of the drawings]

[0015] [Figure 1] Illustrative diagram of lens diffusion characteristics [Figure 2] 1 is a schematic diagram of an image processing apparatus according to a first embodiment of the present invention; [Figure 3] Functional block diagram of an image processing unit according to the first embodiment [Figure 4A] An explanatory diagram of dynamic calculation of a light source determination threshold value [Figure 4B]An explanatory diagram of dynamic calculation of a light source determination threshold value [Figure 5] Functional block diagram of a disparity calculation unit according to the first embodiment [Figure 6] An example of weights set by the weight setting unit of the first embodiment [Figure 7] An example of luminance values ​​in light source and non-light source areas [Figure 8] 1 is a functional block diagram of a disparity matching unit according to a first embodiment; [Figure 9] Another example of a functional block diagram of the disparity matching unit according to the first embodiment [Figure 10] An explanatory diagram of luminance subtraction processing that reflects weights [Figure 11] Functional block diagram of the light source detection unit of the second embodiment [Figure 12] An example of an object identification result used in Example 2 [Figure 13] Functional block diagram of a disparity calculation unit according to a second embodiment [Figure 14] Example of integration weight calculation process [Figure 15] Functional block diagram of an image processing unit according to a third embodiment [Figure 16] 10 is a functional block diagram of a reliability calculation unit according to a third embodiment. [Figure 17] Another example of a functional block diagram of the reliability calculation unit according to the third embodiment [Figure 18] Functional block diagram of a parallax calculation unit according to a fourth embodiment DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment of the image processing device of the present invention will be described with reference to the drawings. Note that, although the present invention will be described below using an image processing device mounted on a vehicle as an example, the image processing device of the present invention is not limited to being mounted on a vehicle. [Example]

[0017] An image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to FIGS.

[0018] First, a schematic configuration of the image processing device 1 of the first embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the image processing device 1 of the first embodiment includes a stereo camera 11, a memory 12, an arithmetic processing unit 13, an image processing unit 14, and an external output unit 15. The components are connected to each other via a communication line 16 so as to be able to communicate with each other.

[0019] The stereo camera 11 is a camera system that uses a left camera 2L and a right camera 2R to synchronously capture left and right images. Hereinafter, any one of the synchronously captured left and right images will be referred to as a base image P0, and the other as a reference image P1. In this embodiment, the left camera 2L and the right camera 2R are positioned so that their line of sight are parallel and their fields of view overlap.

[0020] The memory 12 is a storage device such as a semiconductor memory that temporarily stores the base image P0 and the reference image P1, and stores programs executed by the arithmetic processing unit 13 and the image processing unit .

[0021] The arithmetic processing unit 13 is a calculation device such as a CPU (Central Processing Unit) that implements the arithmetic processing described below in accordance with instructions from the program stored in the memory 12.

[0022] The image processing unit 14 is a calculation device such as a GPU (Graphics Processing Unit) that calculates the parallax p based on the base image P0 and the reference image P1. Note that the specific processing executed by the image processing unit 14 to calculate the parallax p will be described later.

[0023] The external output unit 15 is an interface that transmits the disparity p and the like calculated by the image processing unit 14 to an external device (for example, an ECU (Electronic Control Unit)). The ECU is a control device that controls the drive system, braking system, steering system, and the like of the vehicle, and utilizes the disparity p and the like acquired from the external output unit 15 for object detection, driving path detection, and the like, which are functions of an advanced driver assistance system or an autonomous driving system. These functions may be incorporated as part of the image processing device 1.

[0024] <Image processing unit 14> Next, the image processing unit 14 will be described in detail using the functional block diagram of Fig. 3. As shown in the figure, the image processing unit 14 has a light source region identification unit 141 and a parallax calculation unit 142. Each unit will be described in detail below.

[0025] <<Light source area identification unit 141>> The light source region identification unit 141 receives a pair of images captured by the stereo camera 11 as a base image P0 and a reference image P1, and identifies the region in the received images where the light source is captured (hereinafter referred to as the light source region R). Here, the light source in this embodiment is a term that encompasses not only self-luminous automobile headlights, brake lights, tail lights, street lights, traffic lights, etc., but also objects that emit stronger light toward the camera 2 than the surrounding environment, such as mirrors that reflect strong light.

[0026] One method for identifying the light source region R is to refer to the luminance values ​​of pixels in the image and extract an area where the luminance value is equal to or greater than a threshold as the light source region R. In this case, the brighter the light source, the stronger the diffuse component, and the darker the periphery, the stronger the influence of the diffuse component on parallax matching. Therefore, as shown in FIG. 4A, it is desirable to design the light source region identification unit 141 so that the light source determination threshold increases as the periphery determined from the entire image becomes brighter.

[0027] Figure 4B also shows a simple diagram of the relationship between the brightness of the surroundings and the diffuse light component that affects parallax matching. In this diagram, the brightness of the surroundings increases as you move to the right on the horizontal axis, and the brightness of the light source increases as you move up the vertical axis. As is clear from this diagram, even if the brightness of the light source is the same and it generates the same diffuse light, if the surroundings are bright, the diffuse light component will be blended into the surrounding light and become smaller, so its impact on parallax matching will be relatively small. On the other hand, if the surroundings are dark, the influence of the diffuse light component will be relatively large even if the light source is relatively dark. In other words, by determining the threshold for determining a light source based on the brightness of the surroundings, as shown in Figure 4A, it is possible to efficiently extract only the imaged area of ​​the light source (light source area R) that affects parallax matching.

[0028] <<Parallax calculation unit 142>> The parallax calculation unit 142 weights each pixel in the light source region R identified by the light source region identification unit 141, and calculates the final parallax p by performing parallax matching processing using each weighted pixel.

[0029] 5, the disparity calculation unit 142 includes a weight setting unit 142a and a disparity matching unit 142b. Each unit will be described in detail below.

[0030] The weight setting unit 142a sets a weight for each pixel of the image based on the standard image P0 and the reference image P1 captured by the stereo camera 11 and the light source region R identified by the light source region identifying unit 141.

[0031] The weights set by the weight setting unit 142a will be described with reference to FIG. 6. As illustrated here, in the light source region R0 of the reference image P0, the portion A1 where the light source itself is captured has a particularly high luminance value, while the diffuse light component A2 that causes parallax error is distributed around it at low luminance. For this reason, a method of simply setting weights according to luminance values ​​for this reference image P0 is conceivable, as shown in the graph at the bottom left of FIG. 6. Alternatively, weights may be set by taking into consideration not only luminance values ​​but also luminance gradients. In the graph at the bottom left of FIG. 6, the weights change discretely, but they may also be modeled so that they change smoothly in accordance with smoothly changing luminance values.

[0032] Here, we will explain the weights to be set for each region using an example of a light source region R containing a light source and a non-light source region R' containing objects that are not light sources (e.g., a forest of columnar objects) in an image P captured at night. As shown in FIG. 7, the columnar objects in the non-light source region R' are low-luminance pixels simply due to the low amount of light. However, since these are pixels that we want to actively use in parallax matching, we want to set a large weight. On the other hand, the diffuse light components in the light source region R are pixels with low luminance similar to the columnar object imaging region in the non-light source region R'. However, since these are thought to become noise during parallax matching, we want to set a small weight. In other words, even for pixels with similar low luminance, the optimal weights are different between the light source region R and the non-light source region R'.

[0033] Comparing the characteristics of the light source region R and the non-light source region R', which can be useful in distinguishing which category a low-brightness pixel falls into, reveals the following differences between the two. That is, in the non-light source region R', the brightness changes suddenly at the outer edge of the columnar object depending on whether or not the object is present, so even if the overall brightness value is low, it tends to include a large brightness gradient. On the other hand, the diffuse light component of the light source region R has a small brightness value and a gentle brightness gradient.

[0034] Therefore, taking into account the difference in characteristics between the two, by designing weights that are small only when the brightness value is low and the brightness gradient is small (i.e., only when it is considered to correspond to the diffuse light component within the light source region R), it is possible to avoid the risk of assigning an inappropriately small weight to low brightness areas other than the diffuse light component (for example, objects within the non-light source region R').

[0035] To explain the weight w set for each pixel in this embodiment more specifically, if the luminance value at a certain image coordinate (i, j) is L(i, j) and the luminance gradient is L'(i, j), then the weight w defined by the following equation 1 can be assigned to the image coordinate (i, j). Here, a and b in equation 1 are coefficients that act as scale factors that match the scales of the luminance value and the luminance gradient.

[0036]

number

[0037] The parallax matching unit 142b performs parallax matching while weighting each pixel based on the weight set by the weight setting unit 142a, and outputs a parallax p. The method of reflecting the weighting will be described below.

[0038] 8 shows a functional block diagram of the parallax matching unit 142b, which weights each pixel of the brightness difference calculated prior to the parallax calculation. In this example, after calculating the brightness difference between the base image P0 and the reference image P1, a weight is applied to each brightness difference. Then, the weighted brightness difference is used to calculate the parallax by parallax matching. However, this method does not take into account the weight to be set for each pixel on the reference image P1 side, and therefore cannot fully eliminate the influence of pixels with low weights (diffuse light components).

[0039] As an example of a method for improving the problem shown in FIG. 8, FIG. 9 shows a functional block diagram of a parallax matching unit 142b that performs image transformation on both the base image P0 and the reference image P1 in advance to weight each pixel in both images. In this diagram, the base image P0 and the reference image P1, which are luminance images, are transformed in advance so that the luminance of pixels with low weights has little effect on the difference calculation in the subsequent stage. For example, in a low-weight area, if the luminance values ​​of the corresponding pixels are transformed to be all 0 in the left and right images (base image P0 and reference image P1), the difference calculated during parallax matching within that area will always be 0, and differences in other areas will be given priority.

[0040] An example of the actual conversion procedure is shown in Figure 10. The top graph in the figure is a graph showing the horizontal brightness change in the light source region R0 of the reference image P0. In a method of simply multiplying the brightness of this top graph by a weighting coefficient such as that in Equation 1, an image is generated by multiplying each brightness by the weight set for the region, and then parallax matching is performed. This method is simple, but as shown in the middle graph in Figure 10, the image changes suddenly between areas with strong and weak weights, and new edges are generated in areas that did not originally exist, which can lead to the mistaken belief that an object that did not originally exist has been detected, leading to incorrect matching.

[0041] Therefore, as shown in the lower graph of Figure 10, a luminance image is generated by subtracting a predetermined amount from the luminance of the entire image, and this is used as the target for parallax matching. This allows for parallax matching with minimal impact from weight-changed areas. In this method, matching is performed after a certain luminance value is uniformly subtracted from the entire area identified as the light source. Because luminance is subtracted from bright areas as well, a constant luminance gradient is maintained in high-luminance areas, as shown in the lower graph. The specific value of the luminance to be subtracted may be set so that the luminance values ​​of all pixels with a weight below a certain value become 0, or so that the luminance of the light source area becomes 20% of the initial value. These values ​​are also affected by the diffusion characteristic F of the lens 21 used, so they must be designed based on the parallax accuracy required for the stereo camera 11.

[0042] When the image processing device 1 of the embodiment described above captures an image of a light source and calculates the parallax, it removes the influence of diffused light components around the light source before calculating the parallax, thereby minimizing the parallax error caused by differences in the characteristics of the left and right lenses and achieving highly accurate distance measurement. [Example]

[0043] Second Embodiment Next, an image processing device 1 according to a second embodiment of the present invention will be described with reference to Figures 11 to 14. Note that a duplicated description of points common to the first embodiment will be omitted.

[0044] Example 2 is a modification of Example 1, and is an example in which the object identification result I of an object that can be a light source is acquired from outside. According to this example, the light source region on which the image processing of the present invention should be performed can be specified by utilizing information other than the image luminance value, and image processing can be performed without overlooking important light sources.

[0045] A functional block diagram of the light source region identification unit 141 in this embodiment is shown in Fig. 11. As shown here, the light source region identification unit 141 in this embodiment receives as input the base image P0, the reference image P1, and the object identification result I, and outputs a light source region R, and has a monitoring region setting unit 141a and a brightness change determination unit 141b.

[0046] The object identification result I is a combination of the area in the image where a three-dimensional object is detected and a label indicating the type of the three-dimensional object, and is generated by an object identification unit (not shown) using pattern matching or the like. An example of the object identification result I is shown in FIG. 12. For example, if a vehicle detection result Ob1 is obtained as the object identification result I, there is a possibility that the brake lights or tail lights are on, so the light source area identification unit 141 treats the area indicated by the vehicle detection result Ob1 as a possible light source object. Specifically, the monitoring area setting unit 141a sets an area where the vehicle's brake lights are likely to be present as the monitoring area. Then, the relevant area is tracked in chronological order, and the brightness change determination unit 141b determines the moment when the brightness value changes significantly. If the brightness value changes in the brighter direction, it is determined that this is the timing for the brake lights to be turned on, and the relevant area is output as a light source area.

[0047] Similarly, when a signal detection result Ob2 is obtained as the object identification result I, the monitoring area setting unit 141a sets the position of the signal lamp as a possible light source object in the monitoring area, as described above, and then monitors the brightness change in the area with the brightness change determination unit 141b, and if it becomes brighter, outputs it as a light source area.

[0048] On the other hand, when the other detection result Ob3 of an object that cannot be a light source, such as a roadside tree, is obtained, the monitoring area setting unit 141a does not need to set a particular monitoring area.

[0049] Next, a functional block diagram of the disparity calculation unit 142 of this embodiment is shown in Fig. 13. As shown here, the disparity calculation unit 142 of this embodiment has a type parameter setting unit 142c and a weight merging unit 142d in addition to the weight setting unit 142a and the disparity matching unit 142b described in Fig. 5 of the first embodiment, etc.

[0050] The type parameter setting unit 142c receives the object identification result I and sets a type parameter in addition to the weighting in the first embodiment. The type parameter is a weight that determines the importance of the weighting process of the present invention based on the importance of distance measurement for the target object. When strong weighting is performed, the diffuse light component is effectively removed, improving distance measurement accuracy. On the other hand, the number of pixels not used for parallax matching increases, which reduces the stability of detection. The opposite is true when the weighting is weak. Since these effects are a trade-off, it is important to use the weighting appropriately depending on the scene.

[0051] As shown in FIG. 12, if the type of the object being imaged is known in advance based on the object identification result I obtained from an external source, for example, if the preceding vehicle has its brake lights on, it may be subject to emergency braking by the vehicle itself, and both the urgency and the required accuracy of distance measurement are high. Therefore, the diffused light component according to the present invention should be removed as much as possible to perform high-precision distance measurement. However, if the vehicle has its tail lights on, the urgency is low and the lamps are dimmer than the brake lights, so the amount of error due to the influence of diffused light is smaller, and therefore the removal of the diffused light component can be smaller than that of the brake lights. Furthermore, if the vehicle is a traffic light or street light, the required accuracy of distance measurement is low.

[0052] By changing the weighting according to the type in this way, when distance measurement accuracy is required, this process is performed strongly to prioritize accuracy, and when this is not the case, this process is weakened to ensure stability. The type parameter is output to subsequent processing as a parameter that indicates these tendencies.

[0053] The weights and type parameters calculated by the weight setting unit 142a and type parameter setting unit 142c are integrated into weights to be used for disparity matching by the weight merging unit 142d. For the sake of explanation, an example of integrated weight calculation when the same light source is determined to be the brake light of a preceding vehicle and an example of integrated weight calculation when the same light source is determined to be a traffic light are shown in FIG.

[0054] As shown in the left diagram of Fig. 14, when the light source is determined to be a brake light, the weighting of the light source portion and the diffused light portion is changed more steeply in order to reduce the influence of diffused light as much as possible and improve distance measurement accuracy. This is because, in order to avoid a rear-end collision with a leading vehicle that is suddenly decelerating using control from an advanced driver assistance system or autonomous driving system, it is necessary to accurately estimate the distance to the brake light of the leading vehicle.

[0055] On the other hand, as shown in the right diagram of Figure 14, when the light source is determined to be a signal, the distance measurement accuracy is not improved in the light source area, and the stability of signal detection is maintained by increasing the information used for disparity matching. This is because there is no particular adverse effect even if the distance to the signal cannot be accurately estimated.

[0056] According to this embodiment, the light source area where weighting processing should be performed is identified by utilizing information other than image brightness values, and by performing weighting according to the type of light source, it is possible to perform parallax matching with appropriate removal of diffuse light components only for light sources that require distance measurement accuracy. [Example]

[0057] Next, an image processing device 1 according to a third embodiment of the present invention will be described with reference to Figures 15 to 17. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0058] The third embodiment is a modification of the first embodiment, and further includes a reliability calculation unit 143 that calculates a reliability map M for each parallax in addition to the image processing unit 14 of the first embodiment. According to this embodiment, it is possible to output a reliability map M for each output parallax p, and when using the parallax p in subsequent processing, it is possible to selectively perform high-precision ranging using only the parallax p with high reliability indicated by the reliability map M, or dense point cloud processing using points with low reliability.

[0059] A configuration diagram of the image processing unit 14 in this embodiment is shown in Fig. 15. As shown here, the image processing unit 14 of this embodiment is obtained by adding a reliability calculation unit 143 to the rear end of the image processing unit 14 of embodiment 1 (see Fig. 3). As a result, after the disparity calculation unit 142 calculates the disparity p using the above-mentioned method, the reliability calculation unit 143, which calculates the reliability of each disparity, can calculate the reliability map M.

[0060] 16 shows an example of a functional block diagram of the reliability calculation unit 143. The reliability calculation unit 143 shown here has a normal disparity matching unit 143a and a disparity comparison unit 143b.

[0061] The normal disparity matching unit 143a performs simple disparity calculation without weighting. While the above-mentioned block matching and other methods are available for calculating disparity, the same method as that used by the disparity matching unit 142b of the disparity calculation unit 142 is used, with the only difference being the presence or absence of weighting. Hereinafter, the disparity p calculated here will be referred to as the normal disparity pn and the weighted disparity pw, which is the output of the disparity matching unit 142b, to distinguish them from each other.

[0062] Next, the parallax comparison unit 143b compares the normal parallax pn with the weighted parallax pw for each pixel. Pixels with matching weights are determined to be unaffected by the diffuse light component, and therefore the reliability of the pixel can be determined to be high. On the other hand, if the normal parallax pn and the weighted parallax pw differ, it is determined to be affected by the diffuse light component, and therefore the reliability of the pixel can be determined to be low. Note that the area where the normal parallax pn and the weighted parallax pw differ is an area where the parallax error has been removed by removing the diffuse light component according to the present invention.

[0063] However, as described in the second embodiment, the weighted parallax pw output by the parallax matching unit 142b is equivalent to performing parallax matching by removing pixels containing diffuse light components, and the number of pixels used for parallax matching is smaller than usual, resulting in a lower reliability. Furthermore, since the larger the difference at this time, the greater the influence of the diffuse light components is considered to be, and therefore the reliability is set lower, taking into consideration the possibility that the influence has not been completely removed by weighting. The reliability map M set based on the above idea is used as the final output of the reliability calculation unit 143.

[0064] Alternatively, the reliability may be calculated directly from the diffusion characteristics F of the left and right lenses without comparing the disparity p using the disparity comparison unit 143b. A functional block diagram of the reliability calculation unit 143 in this case is shown in FIG. 17. The reliability calculation unit 143 shown here includes a diffusion estimation unit 143c and a diffusion characteristic-based reliability generation unit 143d. A method for calculating the reliability using the reliability calculation unit 143 in FIG. 17 will be described in detail below.

[0065] First, in a preliminary camera assembly process or the like, an image of a point light source S is actually captured and the diffusion characteristic F of the lens 21 is measured.

[0066] Next, the diffusion degree estimation unit 143c estimates the diffusion degree in each image region of the reference image P0. This diffusion degree is a process of estimating the degree of blur in a small region of the image by looking at the frequency components, brightness gradient, etc. of the small region of the image, and ultimately compares it with the diffusion characteristic F of the lens 21 to determine whether the small region contains a component similar to the diffusion component caused by capturing a light source.

[0067] Then, the diffusion characteristic reliability generation unit 143d determines that in an area having a component similar to the diffusion characteristic F of the lens 21, the parallax calculation is influenced not by a texture due to the presence of an object but by a diffused light component captured pseudo-by diffusion from a light source. In such a region, the reliability is determined to be low. Therefore, the reliability generation unit 143d is designed so that the reliability is lowered as the region becomes more similar to the diffusion characteristic F of the lens 21.

[0068] It is not necessary to select only one of the reliability calculation methods shown in FIG. 16 and FIG. 17, and they may be used simultaneously.

[0069] According to this embodiment, it is possible to extract the parallax p whose reliability has been reduced due to the diffused light component, and to output a reliability map M that indicates the reliability of each parallax p, and it is possible to perform processing using the reliability map M in subsequent processing in the ECU, etc. [Example]

[0070] Next, an image processing device 1 according to a fourth embodiment of the present invention will be described with reference to Fig. 18. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0071] Example 4 is a modification of Example 1, and is an example in which the magnitude of the influence of a light source is determined based on the brightness of the surroundings. According to this example, in a scene in which the influence of a diffused light component is large, such as a strong light source at night, the weighting gradient is increased, and the influence can be appropriately removed.

[0072] A functional block diagram of the disparity calculation unit 142 in this embodiment is shown in Fig. 18. As shown here, the disparity calculation unit 142 in this embodiment is provided with a brightness parameter setting unit 142e instead of the type parameter setting unit 142c in the second embodiment.

[0073] The brightness parameter setting unit 142e of this embodiment receives the reference image P0 as input and outputs a setting value for the brightness parameter. Here, the brightness parameter is a weight that determines the degree of importance of the weighting process of the present invention based on the importance of distance measurement for the target object. The brightness of the surroundings is determined based on the luminance value of the entire screen of the reference image P0. If it is determined that the surroundings are dark, the weight merging unit 142d applies strong weighting and strongly removes the diffuse light component. On the other hand, if it is determined that the surroundings are bright, the diffuse light component is considered to be sufficiently small compared to other light, so the weight merging unit 142d weakens the weighting to prevent a loss of information. The effect of the strength of weighting in the weight merging unit 142d is the same as that of the light source type in the second embodiment.

[0074] Here, we have described an embodiment in which the brightness of the surrounding area is determined from an image. However, if time information can be obtained from an external source, the brightness parameter may be determined from information on whether it is morning, noon, or night. If GPS or map information is available, the brightness parameter may be determined from information such as whether it is inside or outside a tunnel.

[0075] According to this embodiment, weighting is performed according to the degree of influence of the diffuse light component due to changes in ambient brightness, and parallax matching can be performed with the diffuse light component appropriately removed at night, when removal of the diffuse light component is particularly necessary.

[0076] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0077] Furthermore, the above-mentioned components may be partially or entirely configured in hardware, or may be configured to be realized by a processor executing a program. Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0078] 1. Image processing device 11 Stereo Camera 12 Memory 13 Processing unit 14 Image processing section 141 Light source area identification part 141a Monitoring area setting section 141b Luminance change determination unit 142 Parallax calculation unit 142a Weight setting section 142b Parallax matching section 142c Type parameter setting section 142d Weight merging part 142e Brightness parameter setting section 143 Reliability calculation unit 143a Normal parallax matching section 143b Parallax comparison section 143c Diffusion estimation unit 143d Reliability generation unit based on diffusion characteristics 15 External output section 16 Communication Lines 2 Cameras 21 Lens 22 Image sensor 2L Left camera 2R Right Camera S point light source F Diffusion characteristics Ideal image of a Pi point source Pr Actual image of a point light source P0 Reference Image P1 Reference Image p parallax R light source area R' Non-light source area I. Object Identification Results

Claims

1. a light source area specifying unit that specifies an area where a light source is captured in each of a plurality of images simultaneously acquired by a plurality of cameras having overlapping fields of view; a parallax calculation unit that weights pixels in a light source area according to the luminance values ​​of the pixels, and calculates parallax by performing parallax matching of the light source area using the weighted pixels; An image processing device comprising:

2. 2. The image processing device according to claim 1, The image processing device according to claim 1, wherein the parallax calculation unit weights pixels with high luminance values ​​so that they have a greater influence on the parallax matching than pixels with low luminance values.

3. 2. The image processing device according to claim 1, The image processing device is characterized in that the parallax calculation unit performs parallax matching processing after uniformly subtracting luminance values ​​from the entire light source area.

4. 2. The image processing device according to claim 1, The image processing device, wherein the parallax calculation unit weights the luminance values ​​and luminance gradients so that the higher the luminance values ​​and luminance gradients, the greater the influence they have on parallax matching.

5. 2. The image processing device according to claim 1, The image processing device, wherein the parallax calculation unit changes the degree of change in weight depending on the type of light source.

6. 2. The image processing device according to claim 1, The image processing device is characterized in that the light source area specifying unit detects an area in the captured image where the luminance value is equal to or greater than a threshold value.

7. 7. The image processing device according to claim 6, 10. An image processing device, comprising: an image processing unit configured to dynamically change the threshold value so that the threshold value increases as the entire image becomes brighter.

8. 2. The image processing device according to claim 1, The light source area identification unit acquires from an external device an object identification result that identifies an object as having the potential to emit light, monitors the luminance value of the area identified by the object identification result in time series, and determines that a light source attached to a preceding vehicle has emitted light when the luminance value becomes large.

9. 2. The image processing device according to claim 1, The image processing device further comprises a reliability map calculation unit that calculates a reliability map indicating reliability corresponding to each of the parallaxes obtained by the parallax calculation unit.

10. 10. The image processing device according to claim 9, The image processing device is characterized in that the reliability map calculation unit calculates disparity without using the weight, compares it with the disparity calculation result using the weight, and calculates a reliability map in which the greater the difference, the lower the reliability.

11. 10. The image processing device according to claim 9, The reliability map calculation unit calculates a reliability map in which the reliability decreases as the difference in diffusion characteristics between multiple cameras measured in advance increases or the brightness of the target light source increases.

12. 2. The image processing device according to claim 1, The image processing device, wherein the parallax calculation unit includes a brightness parameter setting unit that sets the degree of change in weight to be greater as the surrounding brightness information becomes darker.

13. 13. The image processing device according to claim 12, The image processing device is characterized in that the brightness parameter setting unit determines the brightness parameter using time information or map information obtained from an external source.

14. 13. The image processing device according to claim 12, The image processing device, wherein the brightness parameter setting unit determines the brightness parameter based on a luminance value of a reference image.

15. a light source area specifying step of specifying an area where a light source is captured in each of a plurality of images simultaneously acquired by a plurality of cameras having overlapping fields of view; a parallax calculation step of weighting pixels in a light source area according to luminance values ​​of the pixels, and using the weighted pixels to perform parallax matching of the light source area to obtain parallax; An image processing method comprising:

Images