Image processing apparatus and image processing method

Abstract

The invention provides an image processing device which uses a stereoscopic camera to shoot a light source and eliminates the influence of diffused light components and accurately calculates parallax even if the diffused light components are different due to different characteristics of a left lens and a right lens. In order to achieve the purpose, the present invention is provided with: a light source region specifying unit that specifies an imaging region of a light source in each of a plurality of images simultaneously obtained by a plurality of cameras having overlapped fields of view; and a parallax calculation unit that calculates a parallax by weighting the pixels in the light source region on the basis of the luminance values of the pixels, and performing parallax matching in the light source region using the weighted pixels.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method for calculating the three-dimensional position of a photographed object based on a pair of images synchronously photographed by a stereo camera. Background Art

[0002] In recent years, cars equipped with advanced driver assistance systems or automatic driving systems are becoming more and more popular. These systems estimate the three-dimensional positions of other vehicles, pedestrians, obstacles, etc. around the vehicle, the center line of the road, lane boundaries, and other camera objects, and control the driving, braking, steering, etc. of the vehicle based on these estimation results.

[0003] As one of the external recognition devices used in these systems, a stereo camera is known. In a stereo camera, multiple cameras arranged at different positions synchronously capture the same object from different viewpoints, and the distance to the object is calculated based on the deviation in the way the same object is viewed in the multiple captured images, i.e., the so-called parallax, thereby estimating the three-dimensional position of the captured object.

[0004] When calculating disparity, methods such as block matching are used. This method uses one of a pair of images taken synchronously by a pair of cameras as a reference image, extracts a small area specified on the reference image, and searches for the same small area from the other reference image. To determine whether the small areas are the same, a cost function such as SAD (Sum of Absolute Difference) is used. However, the use of these disparity calculation methods is based on the premise that the optical characteristics of each camera are the same.

[0005] The search in the block matching process is usually performed in units of pixels, but there is a patent document 1 as a technology for improving the ranging resolution through post-processing. In the abstract of the document, it is recorded as a subject that "the reduction in ranging resolution at long distances caused by parallax in units of pixels obtained by processing stereo images is eliminated, and the ranging resolution is improved from short distances to long distances." As a solution, it is recorded that "for a pair of images taken by a stereo camera 10, in the stereo processing unit 30, the city block distance is calculated for each small area of ​​each image and the correlation is obtained, and stereo matching is performed to determine the corresponding small area, and a distance image is generated in which the pixel offset (= parallax) generated according to the distance to the object is imaged as distance data. Furthermore, in the recognition processing unit 40, stereo matching is performed using the data of the reference image and the comparison image, and the parallax (sub-pixel component) of less than 1 pixel is obtained, and the parallax in units of pixels obtained from the range image is interpolated at a resolution of less than 1 pixel. As a result, the reduction in ranging resolution at long distances is eliminated, and the ranging accuracy from short distances to long distances is ensured." Prior art literature Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2000-283753 Summary of the invention Problem that the invention aims to solve

[0007] In order to ensure the ranging accuracy by the technology of Patent Document 1, it is necessary to have the same optical characteristics of the left and right cameras of the stereo camera. However, in reality, the optical characteristics of each camera are not necessarily the same, so when calculating the parallax on the premise that the optical characteristics of each camera are the same, the parallax calculation result contains errors.

[0008] One of the main reasons for the difference in optical characteristics of each camera is the difference in diffusion characteristics F of the lens of each camera. Figure 1 As shown, in a camera 2 having a lens 21 (shown simplified as a single lens, but a combination of multiple lenses may also be used) and an imaging element 22 (CMOS image sensor, etc.), light emitted from an ideal point light source S is projected onto the imaging element 22 through the lens 21 to generate an image P.

[0009] In this case, since a point light source S having no size is imaged, ideally, as shown in the image Pi of the "ideal model", the desired light source is imaged as only one pixel of the point light source S on the image pickup element 22. However, in reality, the light of the point light source S is diffused to the periphery of the imaging position according to the diffusion characteristic F of the lens 21, and therefore, as shown in the image Pr of the "actual model", the light from the point light source S is imaged over a wide range with the brightness gradually attenuated toward the periphery.

[0010] Therefore, if the diffusion characteristics F of the left and right cameras of a stereo camera are different, even if it is assumed that a point light source S of the same brightness is photographed from the same position, different images will be photographed by each camera. As a result, when calculating parallax using stereo cameras with different diffusion characteristics F of each camera, since block matching is performed based on different diffusion components of each image, the parallax of the light source cannot be accurately calculated, and the three-dimensional position of the light source cannot be accurately estimated. In particular, when the light source is located far away, due to the characteristics of the stereo camera, even a small parallax error will lead to a large ranging error. Therefore, in the process of measuring the distance of a distant object and using it for the control of the subsequent stage, for example, in the function of following the preceding vehicle using the on-board stereo camera, there is a problem of large parallax errors caused by the difference in diffusion characteristics F of each camera.

[0011] As a solution to this problem, it is considered to pair the lenses in such a way that the left and right cameras of the stereo camera have the same diffusion characteristic F. However, even for the same batch of lenses with the same design values, there are deviations in the diffusion characteristic F. Therefore, in order to pair lenses with consistent diffusion characteristics, it is necessary to measure the diffusion characteristic F of each lens, find lenses with similar characteristics and pair them, which is a lot of work.

[0012] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing device and an image processing method capable of accurately calculating parallax even if the diffusion characteristics of the cameras constituting a stereo camera are different. Technical means of solving problems

[0013] In order to achieve the above-mentioned purpose, the present invention comprises: a light source area determination unit, which determines the shooting area of ​​the light source in each of a plurality of images simultaneously obtained by a plurality of cameras with overlapping fields of view; and a parallax calculation unit, which weights the pixel according to the brightness value of the pixel in the light source area, and uses the weighted pixels to perform parallax matching of the light source area, thereby calculating the parallax. Effects of the Invention

[0014] According to the image processing device and the image processing method of the present invention, even if the diffusion characteristics of the cameras constituting the stereo camera are different, it is possible to accurately calculate the parallax. Other problems, configurations, and effects than those described above will be clarified by the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram to explain the diffusion characteristics of a lens. Figure 2 This is a schematic diagram of the structure of the image processing device of Example 1. Figure 3 This is a functional block diagram of the image processing unit of the first embodiment. Figure 4A This is a diagram for explaining the dynamic calculation of the light source determination threshold. Figure 4B This is a diagram for explaining the dynamic calculation of the light source determination threshold. Figure 5 This is a functional block diagram of the parallax calculation unit of the first embodiment. Figure 6 This is an example of the weight set by the weight setting unit of the first embodiment. Figure 7 This is an example of the brightness value of the light source area and the non-light source area. Figure 8 This is an example of a functional block diagram of the parallax matching unit of Example 1. Fig. 9 This is another example of the functional block diagram of the parallax matching unit of the first embodiment. Fig.10 This is an explanatory diagram of the brightness subtraction process that reflects the weight. Fig.11 This is a functional block diagram of the light source detection unit of Example 2. Fig.12 This is an example of the object recognition result used in Example 2. Fig.13 This is a functional block diagram of a parallax calculation unit in the second embodiment. Fig.14 This is an example of comprehensive weight calculation processing. Fig.15 This is a functional block diagram of the image processing unit of the third embodiment. Fig.16 This is an example of a functional block diagram of a reliability calculation unit of the third embodiment. Fig.17 This is another example of the functional block diagram of the reliability calculation unit of the third embodiment. Fig.18 This is a functional block diagram of a parallax calculation unit in the fourth embodiment. DETAILED DESCRIPTION

[0016] Hereinafter, embodiments of the image processing device of the present invention will be described with reference to the drawings. In addition, the present invention will be described below by taking an image processing device mounted on a vehicle as an example, but the image processing device of the present invention is not limited to being mounted on a vehicle. Example 1

[0017] use Figures 2 to 10 An image processing device 1 according to a first embodiment of the present invention will be described.

[0018] First, use Figure 2 The schematic configuration of the image processing device 1 of the first embodiment is described. As shown here, the image processing device 1 of the present embodiment includes a stereo camera 11, a memory 12, a calculation processing unit 13, an image processing unit 14, and an external output unit 15. The components are connected to each other via a communication line 16 so as to be able to communicate with each other.

[0019] The stereo camera 11 is a camera system that uses a left camera 2L and a right camera 2R to synchronously shoot a left image and a right image. Hereinafter, either one of the synchronously shot left and right images is referred to as a base image P0, and the other is referred to as a reference image P1. In addition, the left camera 2L and the right camera 2R of this embodiment are arranged so that their sight lines are parallel and their fields of view overlap.

[0020] The memory 12 is a storage device such as a semiconductor memory that temporarily stores the standard image P0 and the reference image P1 and stores programs executed by the arithmetic processing unit 13 and the image processing unit 14 .

[0021] The calculation processing unit 13 is a calculation device such as a CPU (Central Processing Unit), and implements the calculation processing described below according to the instructions of the program stored in the memory 12 .

[0022] The image processing unit 14 is a computing device such as a GPU (Graphics Processing Unit) that calculates the parallax p based on the standard image P0 and the reference image P1. The specific processing for calculating the parallax p performed by the image processing unit 14 will be described later.

[0023] The external output unit 15 is an interface for sending the parallax p etc. calculated by the image processing unit 14 to the outside (for example, ECU (Electronic Control Unit)). In addition, the ECU is a control device for controlling the driving system, braking system, steering system, etc. of the vehicle, and the parallax p etc. obtained from the external output unit 15 is used for object detection or driving path detection etc., which is a function of an advanced driving assistance system or an automatic driving system. In addition, these functions can also be incorporated as part of the image processing device 1.

[0024] <Image Processing Unit 14> Next, use Figure 3 The image processing unit 14 is described in detail with reference to a functional block diagram of FIG. 1. As shown in the figure, the image processing unit 14 includes a light source region determination unit 141 and a parallax calculation unit 142. The details of each unit will be described in sequence below.

[0025] <<Light Source Region Determining Unit 141>> The light source region determination unit 141 receives a pair of images captured by the stereo camera 11 as the base image P0 and the reference image P1, and determines the shooting region of the light source (hereinafter referred to as the light source region R) in the received image. Here, the light source in this embodiment refers to not only the headlights, brake lights, tail lights, street lights, and signals of self-luminous automobiles, but also objects such as mirrors that reflect strong light and emit stronger light to the camera 2 than the surrounding environment.

[0026] As one method of determining the light source region R, the brightness value of the pixels in the reference image is considered, and the region with a brightness value above a threshold is extracted as the light source region R. In addition, at this time, the brighter the light source, the stronger the diffusion component, and the darker the surrounding area, the stronger the influence of the diffusion component on the parallax matching. Therefore, if Figure 4A As shown, it is preferable to design the light source region identification unit 141 so that the brighter the brightness of the surrounding area determined based on the entire image, the larger the light source determination threshold.

[0027] in addition, Figure 4BThis simply shows the relationship between the brightness of the surroundings and the diffuse light component that affects parallax matching. In this figure, the brighter the surroundings are as you move to the right on the horizontal axis, and the brighter the light source is as you move to the top on the vertical axis. As can be seen from this figure, even if the brightness of the light source is the same and produces the same diffuse light, if the surroundings are bright, the diffuse light component is mixed with the surrounding light and becomes smaller, so the impact on parallax matching becomes relatively small. On the other hand, if the surroundings are dark, the impact of the diffuse light component becomes relatively large even with a relatively dark light source. That is, if Figure 4A As shown, by determining the threshold value for determining a light source based on the brightness of the surrounding area, it is possible to efficiently extract only the imaging area of ​​the light source (light source area R) that affects the parallax matching.

[0028] <<Disparity Calculation Unit 142>> The parallax calculation unit 142 weights each pixel of the light source region R determined by the light source region determination unit 141 , and calculates a final parallax p by performing a parallax matching process using each weighted pixel.

[0029] like Figure 5 As shown in the functional block diagram of FIG. 1 , the parallax calculation unit 142 includes a weight setting unit 142a and a parallax matching unit 142b. The details of each unit will be described in turn below.

[0030] The weight setting unit 142 a sets weights for each pixel of the image based on the standard image P0 and the reference image P1 captured by the stereo camera 11 and the light source region R determined by the light source region determination unit 141 .

[0031] use Figure 6 The weights set by the weight setting unit 142a are described below. As shown in the example here, in the light source area R0 of the reference image P0, the brightness value of the portion A1 that captures the light source itself is particularly high, and the diffuse light component A2 that is the main cause of the parallax error is distributed around it with low brightness. Figure 6 As shown in the lower left figure, for the reference image P0, a method of setting weights simply based on the brightness value can be considered. Alternatively, the weights can be set by considering not only the brightness value but also the brightness gradient. Figure 6 In the lower left figure, the weights change discretely, but it is also possible to model the weights in a way that changes smoothly in contrast to the brightness values ​​that change smoothly.

[0032] Here, in an image P taken at night, the light source area R including the light source and the non-light source area R' including objects that are not light sources (for example, a forest of columnar objects) are used as examples to explain the weights that should be set for each area. Figure 7As shown, the numerous columnar objects in the non-light source area R' become low-brightness pixels simply because of the small amount of light, but since they are pixels that are intended to be actively used for parallax matching, the weights set should be increased. On the other hand, the diffuse light component of the light source area R becomes a pixel of the same degree of low brightness as the columnar object imaging area in the non-light source area R', but since it is considered to be noise during parallax matching, the weight set should be reduced. That is, even for pixels of the same degree of low brightness, the optimal weights are different in the light source area R and the non-light source area R'.

[0033] Low-brightness pixels can be used as a reference when distinguishing how to classify. When comparing the characteristics of the light source area R and the non-light source area R', the following differences can be seen between the two. That is, in the non-light source area R', since the brightness changes sharply at the outer edge of the columnar object depending on the presence or absence of the object, there is a tendency to include a large brightness gradient even if the overall brightness value is small. On the other hand, the brightness value of the diffuse light component of the light source area R is small, and the brightness gradient is also gentle.

[0034] Therefore, in view of the difference in characteristics between the two, by designing weights that become smaller only when the brightness value is low and the brightness gradient is small (that is, only when it is considered to be equivalent to the diffuse light component within the light source area R), the risk of inappropriately reducing the weight of low-brightness parts other than the diffuse light component (for example, objects within the non-light source area R') can be avoided.

[0035] In this embodiment, if the weight w set for each pixel is described in more detail, assuming that the brightness value at a certain image coordinate (i, j) is L(i, j) and the brightness gradient is L'(i, j), the weight w specified by the following formula 1 can be given to the image coordinate (i, j). Here, a and b in formula 1 are coefficients that function as a scaling factor that makes the ratio of the brightness value and the brightness gradient consistent.

[0036] [Formula 1] w=MAX(aL(i,j),bL′(i,j))…(Formula 1)

[0037] The parallax matching section 142 b performs parallax matching while weighting each pixel based on the weight set by the weight setting section 142 a , and outputs the parallax p. A weighting reflection method will be described below.

[0038] Figure 8The functional block diagram of the parallax matching unit 142b is shown, which performs weighting for each pixel on the brightness difference calculated before the parallax. In this example, after calculating the brightness difference between the standard image P0 and the reference image P1, weighting is applied to each brightness difference. Then, the parallax is calculated by parallax matching using the brightness difference to which the weight is applied. However, in this method, since the weight to be set for each pixel on the reference image P1 side is not considered, the influence of pixels with low weights (diffused light components) cannot be fully eliminated.

[0039] Therefore, as an improvement Figure 8 An example of a problem approach, Fig. 9 The functional block diagram of the parallax matching unit 142b is shown, and the parallax matching unit 142b realizes weighting of each pixel of the two images by performing image conversion on both the base image P0 and the reference image P1 in advance. In this figure, the base image P0 and the reference image P1, which are brightness images, are transformed in advance so that the brightness of pixels with small weights has a smaller impact in the differential calculation of the subsequent stage. For example, in an area with a low weight, if the brightness value converted to the pixel is 0 in both the left and right images (base image P0 and reference image P1), then in this area, the difference calculated during parallax matching is always 0, and the differences in other areas are processed preferentially.

[0040] Fig.10 An example of the actual conversion process is shown. The upper graph of the figure is a graph showing the lateral brightness change of the light source area R0 of the reference image P0. In the method of simply multiplying the weight coefficient of Formula 1 or the like by the brightness of the upper graph, an image is generated by multiplying the weight set in the area by each brightness, and parallax matching is performed. Although this method is simple, Fig.10 As shown in the middle graph of , the image changes dramatically between the parts where the weight is increased and the parts where the weight is decreased, new edges are generated in parts that did not originally exist, and there is also the possibility of mistaking the detection of objects that did not originally exist and causing incorrect matching.

[0041] Therefore, if Fig.10As shown in the lower diagram of , by generating a brightness image obtained by subtracting a specified amount from the brightness of the entire image and using it as the processing object of parallax matching, parallax matching that minimizes the influence of the weight change part can be performed. In this method, a certain brightness value is uniformly subtracted from the entire area determined as the light source and then matched. Since the brightness is also subtracted from the bright part, the brightness gradient remains constant in the high-brightness area as shown in the lower diagram. Regarding the specific numerical value of the subtracted brightness, the subtraction value can be set in such a way that the brightness values ​​of pixels with a weight below a certain value are all 0, or the subtraction value can be set in such a way that the brightness of the light source area is 20% of the initial state. Since they are also affected by the diffusion characteristic F of the lens 21 used, they need to be designed according to the parallax accuracy required by the stereo camera 11.

[0042] By using the image processing device 1 of the embodiment described above, when photographing a light source and performing parallax calculation, the parallax calculation is performed on the basis of eliminating the influence of the diffuse light component around the light source. Therefore, the parallax error caused by the difference in the characteristics of the left and right lenses can be suppressed to a minimum, thereby achieving high-precision ranging. Example 2

[0043] Next, use Figures 11 to 14 An image processing device 1 according to a second embodiment of the present invention will be described. Note that duplicate descriptions of points common to the first embodiment will be omitted.

[0044] Embodiment 2 is a variation of Embodiment 1, and is an embodiment of the case where an object recognition result I of an object that can be a light source is obtained from the outside. According to this embodiment, information other than the image brightness value can be used to determine the light source area to be subjected to the image processing of the present invention, and image processing can be performed without missing important light sources.

[0045] Fig.11 The functional block diagram of the light source region determination unit 141 of the present embodiment is shown. As shown here, the light source region determination unit 141 of the present embodiment takes the base image P0, the reference image P1, and the object recognition result I as inputs, and outputs the light source region R. The light source region determination unit 141 includes a monitoring region setting unit 141a and a brightness change determination unit 141b.

[0046] The object recognition result I is a combination of a region where a three-dimensional object is detected and a label indicating the category of the three-dimensional object on an image, and is generated by an object recognition unit (not shown) using pattern matching or the like. Fig.12An example of an object recognition result I is shown. For example, when a vehicle detection result Ob1 is obtained as the object recognition result I, there is a possibility that the brake lights or tail lights are on. Therefore, in the light source area determination unit 141, the area shown in the vehicle detection result Ob1 is processed as a light source possible object. Specifically, the monitoring area setting unit 141a sets the area where the brake lights of the vehicle may exist as the monitoring area. Then, the area is tracked in a time series, and the brightness change determination unit 141b determines the moment when the brightness value changes greatly. When the brightness value changes in a bright direction, it is determined that the brake lights are on, and the area is output as a light source area.

[0047] Similarly, when the signal detection result Ob2 is obtained as the object recognition result I, as described above, after the monitoring area setting unit 141a sets the signal light position as a possible light source object as the monitoring area, the brightness change judgment unit 141b monitors the brightness change of the area, and when it becomes brighter, it is output as a light source area.

[0048] On the other hand, when another detection result Ob3 of an object that cannot serve as a light source, such as a roadside tree, is obtained, the monitoring area setting unit 141a does not need to particularly set a monitoring area.

[0049] Next, Fig.13 FIG. 1 is a functional block diagram of the parallax calculation unit 142 of the present embodiment. As shown here, the parallax calculation unit 142 of the present embodiment has the following features in addition to the features of the first embodiment: Figure 5 In addition to the weight setting unit 142a and the parallax matching unit 142b described in the above, the EMBODIMENT 142 further includes a type parameter setting unit 142c and a weight integration unit 142d.

[0050] The category parameter setting unit 142c receives the object recognition result I and sets the category parameter differently from the weight of Example 1. The category parameter is a weight that determines the severity of the weighted processing of the present invention according to the importance of ranging the object. When the weighting is strengthened, the effect of removing the diffuse light component is high, so the ranging accuracy is improved. On the other hand, since the number of pixels not used in the parallax matching increases, the stability of the detection is poor. The opposite is true when the weighting is weakened. Since these effects are trade-offs, it is important to use them appropriately according to the scene.

[0051] like Fig.12As illustrated, when the category of the object to be photographed is known in advance based on the object recognition result I obtained from the outside, for example, if it is the vehicle in front with the brake lights on, it may become the emergency braking target of the vehicle, and the urgency and ranging accuracy required are high. Therefore, the diffuse light component of the present invention should be removed as much as possible to perform high-precision ranging. However, if it is a vehicle with the tail lights on, the urgency is low, and since the lights are darker than the brake lights, the error caused by the influence of the diffuse light is also small, so the removal of the diffuse light component can be smaller than that of the brake lights. In addition, if it is a signal or street light, the ranging accuracy required is low.

[0052] Thus, by changing the weight according to the category, when the ranging accuracy is required, the present process is strengthened to give priority to the accuracy, and when the ranging accuracy is not required, the present process is weakened to ensure stability. The category parameter is output to the subsequent processing as a parameter representing these tendencies.

[0053] The weights and category parameters calculated by the weight setting unit 142a and the category parameter setting unit 142c are finally integrated into the weights for parallax matching in the weight merging unit 142d. Fig.14 An example of comprehensive weight calculation when a completely identical light source is determined to be a brake light of a preceding vehicle and an example of comprehensive weight calculation when a completely identical light source is determined to be a signal are shown.

[0054] like Fig.14 As shown in the left figure, when the light source is determined to be a brake light, the weight change between the light source part and the diffuse light part is changed to be more drastic in order to minimize the influence of diffuse light and improve the distance measurement accuracy. This is because in order to avoid rear-ending the front vehicle that is decelerating rapidly through the control of the advanced driver assistance system or the automatic driving system, as a prerequisite, it is necessary to accurately estimate the distance to the brake lights of the front vehicle.

[0055] On the other hand, Fig.14 As shown in the right figure, when a light source is determined to be a signal, the accuracy of distance measurement is not improved in the light source part, and the information used for parallax matching is increased to maintain the stability of signal detection. This is because even if the distance to the signal cannot be correctly estimated, there is no particular adverse effect.

[0056] According to this embodiment, by utilizing information other than image brightness values ​​to determine the light source area to be weighted and performing weighting corresponding to the type of the light source, parallax matching can be performed only on the light source required by the distance measurement system with diffuse light components appropriately removed. Example 3

[0057] Next, use Figure 15 to Figure 17 An image processing device 1 according to a third embodiment of the present invention will be described. Note that duplicate descriptions of points common to the above-described embodiments will be omitted.

[0058] Embodiment 3 is a variation of Embodiment 1, in which a reliability calculation unit 143 for calculating a reliability map M for each parallax is added to the image processing unit 14 of Embodiment 1. According to this embodiment, the reliability map M for each output parallax p can be output, and when the parallax p is used in the post-processing, high-precision distance measurement using only the parallax p with high reliability shown in the reliability map M, dense point group processing using points with low reliability, etc. can be selectively performed.

[0059] Fig.15 The image processing unit 14 of the present embodiment is shown in FIG. Figure 3 ) is added to the back end of the reliability calculation unit 143. Thus, after the parallax calculation unit 142 calculates the parallax p by the above method, the reliability calculation unit 143 that calculates the reliability of each parallax can calculate the reliability map M.

[0060] Fig.16 2 shows an example of a functional block diagram of the reliability calculation unit 143. The reliability calculation unit 143 shown here includes a normal parallax matching unit 143a and a parallax comparing unit 143b.

[0061] The normal parallax matching unit 143a performs a simple parallax calculation without weighting. Although the above-mentioned block matching and other methods are used for parallax calculation, the same method as the parallax matching unit 142b of the parallax calculation unit 142 is used, and the only difference is whether weighting is performed. Hereinafter, the parallax p calculated here is referred to as the normal parallax pn, and is distinguished from the weighted parallax pw which is the output of the parallax matching unit 142b.

[0062] Next, in the parallax comparison unit 143b, the normal parallax pn and the weighted parallax pw are compared for each pixel. Since it is determined that there is no influence of the diffuse light component in the pixel with the same weight, it can be determined that the reliability of the pixel is high. On the other hand, when the normal parallax pn and the weighted parallax pw are different, it is determined that there is an influence of the diffuse light component, so it can be determined that the reliability of the pixel is low. In addition, the area where the normal parallax pn and the weighted parallax pw are different is the area where the parallax error is removed by removing the diffuse light component of the present invention.

[0063] However, as described in Example 2, the weighted disparity pw as the output of the disparity matching unit 142b is equivalent to performing disparity matching by removing pixels containing diffuse light components, and the number of pixels used in the disparity matching is smaller than usual, so the reliability is reduced. At the same time, considering that the larger the difference at this time, the greater the influence of the diffuse light component, the reliability is set lower considering the possibility that the influence is not completely removed by weighting. The reliability map M set based on the above idea is used as the final output of the reliability calculation unit 143.

[0064] Alternatively, the reliability may be directly calculated based on the diffusion characteristics F of the left and right lenses without comparing the parallax p by the parallax comparing unit 143 b. Fig.17 The functional block diagram of the reliability calculation unit 143 at this time is shown. The reliability calculation unit 143 shown here includes a diffusion degree estimation unit 143c and a diffusion characteristic-based reliability generation unit 143d. Fig.17 The reliability calculation method of the reliability calculation unit 143.

[0065] First, in a preliminary camera assembly process or the like, a point light source S is actually photographed, and the diffusion characteristic F of the lens 21 is measured.

[0066] Next, the diffusion estimating unit 143c estimates the diffusion in each image area of ​​the reference image P0. The diffusion is a process of estimating the degree of blur of a small area of ​​the image by observing the frequency component, brightness gradient, etc. of the small area of ​​the image, and finally comparing it with the diffusion characteristic F of the lens 21 to determine whether the small area contains a component similar to the diffusion component caused by the imaging of the light source.

[0067] Furthermore, in the reliability generating unit 143d based on the diffusion characteristic, it is determined that in the area having a component similar to the diffusion characteristic F of the lens 21, the diffuse light component that is simulated and photographed due to the diffusion of the light source rather than the texture caused by the existence of the object affects the parallax calculation. That is, in such an area, it is determined that the reliability is low. Therefore, the reliability generating unit 143d is designed so that the more the area is similar to the diffusion characteristic F of the lens 21, the lower the reliability.

[0068] Fig.16 and Fig.17 There is no need to select only one of the reliability calculation methods, and they can be used simultaneously.

[0069] According to this embodiment, parallax p whose reliability is reduced due to diffuse light components can be extracted, and a reliability map M indicating the reliability of each parallax p can be output, and processing using the reliability map M can be executed in subsequent processing such as an ECU. Example 4

[0070] Next, use Fig.18 An image processing device 1 according to a fourth embodiment of the present invention will be described. Note that duplicate descriptions of points common to the above-described embodiments will be omitted.

[0071] Embodiment 4 is a variation of Embodiment 1, and is an embodiment in which the magnitude of the influence of the light source is determined based on the surrounding brightness. According to this embodiment, in scenes where the influence of diffuse light components is large, such as strong light sources at night, the weighted gradient can be increased to appropriately remove the influence.

[0072] Fig.18 2 is a functional block diagram of the parallax calculation unit 142 of the present embodiment. As shown here, the parallax calculation unit 142 of the present embodiment is provided with a brightness parameter setting unit 142e instead of the category parameter setting unit 142c of the second embodiment.

[0073] The brightness parameter setting unit 142e of this embodiment receives the reference image P0 as input and outputs the setting value of the brightness parameter. Here, the brightness parameter is a weight that determines the severity of the weighted processing of the present invention according to the importance of the distance measurement of the target object. Then, the brightness of the surrounding is determined based on the brightness value of the entire screen of the reference image P0. When it is determined that the surrounding is dark, the weight merging unit 142d strengthens the weighting and removes the diffuse light component. On the other hand, when it is determined that the surrounding is bright, it is considered that the diffuse light component is small enough relative to other lights, so the weight merging unit 142d weakens the weighting to prevent the reduction of information. The effect brought about by the strength of the weighting in the weight merging unit 142d is the same as the light source category in Example 2.

[0074] Here, an embodiment of determining the brightness of the surroundings based on an image is described, but for example, when time information can be obtained from the outside, the brightness parameter can also be determined based on early day and night information. When GPS or map information is available, the brightness parameter can also be determined based on information such as inside and outside the tunnel.

[0075] According to this embodiment, weighting that matches the degree of influence of diffuse light components caused by changes in peripheral brightness can be performed, and parallax matching that appropriately removes diffuse light components can be performed at night when removal of diffuse light components is particularly necessary.

[0076] In addition, the present invention is not limited to the above-mentioned embodiments, and includes various modified examples. For example, the above-mentioned embodiments are embodiments that are described in detail in order to easily explain the present invention, and are not necessarily limited to embodiments having all the structures described. In addition, a part of the structure of one embodiment can be replaced with the structure of another embodiment, and the structure of another embodiment can be added to the structure of one embodiment. In addition, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0077] In addition, each of the above-mentioned structures may be partially or entirely constituted by hardware, or may be constituted to be realized by executing a program by a processor. In addition, the control lines and information lines represent the lines that are considered necessary for the description, and not all control lines and information lines may be represented in the product. In fact, it can be considered that almost all the structures are connected to each other. Explanation of symbols

[0078] 1… Image processing device, 11… Stereo camera, 12… Memory, 13… Calculation processing unit, 14… Image processing unit, 141… Light source area determination unit, 141a… Monitoring area setting unit, 141b… Brightness change determination unit, 142… Parallax calculation unit, 142a… Weight setting unit, 142b… Parallax matching unit, 142c… Category parameter setting unit, 142d… Weight merging unit, 142e… Brightness parameter setting unit, 143… Reliability calculation unit, 143a… Normal parallax matching unit, 1 43b…parallax comparison unit, 143c…diffusion estimation unit, 143d…reliability generation unit based on diffusion characteristics, 15…external output unit, 16…communication line, 2…camera, 21…lens, 22…imaging element, 2L…left camera, 2R…right camera, S…point light source, F…diffusion characteristics, Pi…ideal image of point light source, Pr…actual image of point light source, P0…reference image, P1…reference image, p…parallax, R…light source area, R'…non-light source area, I…object recognition result.

Claims

1. An image processing device, characterized in that: have: a light source region determining unit that determines a photographing region of a light source in each of a plurality of images simultaneously obtained by a plurality of cameras having overlapping fields of view; as well as The parallax calculation unit weights the pixels in the light source area according to the brightness values ​​of the pixels, and performs parallax matching of the light source area using the weighted pixels, thereby obtaining the parallax.

2. The image processing device according to claim 1, characterized in that The parallax calculation unit weights pixels with high luminance values ​​so that the pixels with high luminance values ​​have a greater influence on the parallax matching than the pixels with low luminance values.

3. The image processing device according to claim 1, characterized in that The parallax calculation unit performs parallax matching processing after uniformly subtracting the brightness value of the entire light source area.

4. The image processing device according to claim 1, characterized in that: The disparity calculation unit performs weighting so that the higher the brightness value and the brightness gradient, the greater the influence on the disparity matching.

5. The image processing device according to claim 1, characterized in that: The parallax calculation unit changes the degree of change of the weight according to the type of the light source.

6. The image processing device according to claim 1, characterized in that: The light source region identification unit detects a region having a brightness value greater than or equal to a threshold value in the captured image.

7. The image processing device according to claim 6, characterized in that: The threshold value changes dynamically in such a manner that the brighter the overall image is, the higher the threshold value is.

8. The image processing device according to claim 1, characterized in that: The light source area determination unit obtains an object recognition result of an object recognized as having the possibility of emitting light from the outside, monitors the brightness value of the area determined by the object recognition result in a time series, and determines that the light source installed on the preceding vehicle is emitting light when the brightness value increases.

9. The image processing device according to claim 1, characterized in that: The device further includes a reliability map calculation unit that calculates a reliability map indicating the reliability corresponding to each parallax obtained by the parallax calculation unit.

10. The image processing device according to claim 9, characterized in that: The reliability map calculation unit performs parallax calculation without using the weights, compares the result of parallax calculation using the weights, and calculates a reliability map in which the higher the difference, the lower the reliability.

11. The image processing device according to claim 9, characterized in that: The reliability map calculation unit calculates a reliability map in which the reliability decreases as the difference in diffusion characteristics of a plurality of cameras measured in advance increases or the brightness of a target light source increases.

12. The image processing device according to claim 1, characterized in that: The parallax calculation unit includes a brightness parameter setting unit that sets the brightness parameter based on surrounding brightness information so that the darker the surrounding brightness, the greater the degree of change in the weight.

13. The image processing device according to claim 12, characterized in that: The brightness parameter setting unit determines the brightness parameter using time information or map information acquired from the outside.

14. The image processing device according to claim 12, characterized in that: The brightness parameter setting unit determines a brightness parameter based on a brightness value of a reference image.

15. An image processing method, characterized in that: have: a light source area determining step of determining a shooting area of ​​the light source in each of a plurality of images simultaneously obtained by a plurality of cameras having overlapping fields of view; as well as The parallax calculation step weights the pixels in the light source area according to their brightness values, and uses the weighted pixels to perform parallax matching in the light source area, thereby obtaining the parallax.

Images