Image processing system

Abstract

To enable suppression of the deterioration in the accuracy of distance calculations with a subject even when striation appears in a picked-up image.SOLUTION: One or more processors (CPU 11, GPU 15) identify a high-brightness region, which contains a single light source or glossy surface along with striations extending from the light source or glossy surface, in a pair of picked-up images by a stereo camera 20. Furthermore, the CPU 11 and GPU 15 perform pixel-to-pixel correspondence for each high-brightness region. Next, the CPU 11 and GPU 15 determine the disparity distribution within the high-brightness regions based on the correspondence. Furthermore, the CPU 11 and GPU 15 calculate the distance to the light source or glossy surface based on the disparity distribution.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] Disclosed herein is an image processing device.

[0002] A stereo camera system is mounted on a vehicle to measure the distance between the vehicle and objects around the vehicle. The stereo camera system includes, for example, a pair of cameras.

[0003] The fields of view of a pair of cameras overlap at least partially. Correspondence estimation is performed to find which pixels of each camera correspond to a point on an object. Once a pair of corresponding pixels is found, the horizontal displacement between the two pixels is calculated. This displacement is called disparity. Using the principles of triangulation, the distance from the camera to the object can be calculated from the disparity, the distance between the cameras (baseline), and the focal length of the cameras.

[0004] When a light source is photographed using a stereo camera, differences in the diffuse light components may occur due to differences in the lens characteristics of the left and right cameras. In Patent Document 1, pixels in the light source area are weighted according to their luminance values. The lower the luminance value (the farther from the light source), the smaller the weighting value. In other words, the influence of the diffuse light components is reduced.

[0005] In Patent Document 2, lens characteristics, brightness characteristics, and temperature characteristics are recorded when the camera is manufactured, and the distance to the object is corrected based on these characteristics.

[0006] Furthermore, depending on the aperture settings of the camera, streaks of light called "striations" may appear in the captured image. The striations extend radially from the light source. The direction and length of the striations vary depending on the camera's imaging conditions. In other words, the striations can become noise components in the parallax calculation process. Therefore, in Patent Document 3, if striations are detected in the captured image, the glass surface is determined to be dirty. A message urging the driver to clean the glass surface is then displayed. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-7678 [Patent Document 2] International Publication No. 2023 / 175708 [Patent Document 3] Japanese Patent Application Laid-Open No. 2017-58949 Summary of the Invention [Problem to be solved by the invention]

[0008] This specification discloses an image processing device that can suppress a decrease in accuracy in calculating the distance to a subject even when streaks appear in a captured image. [Means for solving the problem]

[0009] This specification discloses an image processing device. The device includes a stereo camera and a processor. The stereo camera is mounted on a vehicle. The stereo camera captures images of the outside of the vehicle. One or more processors are provided. The one or more processors perform image processing on a pair of images captured by the stereo camera. The one or more processors identify high-brightness areas from each captured image. The high-brightness areas are image areas from which a single light source or shiny surface and light striations are extracted. The light striations extend from the light source or shiny surface. The one or more processors further associate pixels with each other for each high-brightness area. Next, the one or more processors calculate a disparity distribution in the high-brightness area based on the association. The one or more processors further calculate a distance to the light source or shiny surface based on the disparity distribution.

[0010] Variations in the parallax distribution in high-luminance areas may occur. Variations in the parallax distribution mean that a single subject, a light source or a shiny surface, exists in a scattered manner in the front-to-back direction. The main cause of such a disparity in the parallax distribution generated for a single light source or shiny surface is thought to be light striations in high-luminance areas. The effect of light striations can be reduced by determining the parallax distribution and verifying whether or not there is any variation in the distribution.

[0011] In the above configuration, when the disparity distribution is a multi-modal distribution having three or more peak values, the one or more processors exclude the peak values ​​at both ends of the disparity distribution, and calculate the distance to the light source or the glossy surface based on the disparity distribution after the exclusion.

[0012] When three peaks appear in the disparity distribution, some parts of the light source are too close, some parts are too far away, and the rest are in between. The main reason for this unrealistic disparity distribution for a single light source or glossy surface is thought to be due to striations in high-brightness areas. Furthermore, as will be described later, when striations appear in the captured image, peaks appear at both ends of the disparity distribution. Therefore, excluding these peak values ​​reduces the effect of striations.

[0013] In the above configuration, the one or more processors may calculate a mean value, a median value, and a peak mean value in the disparity distribution before excluding the peak values ​​at both ends. The peak mean value is an average of multiple peak values. The one or more processors calculate reliability for the mean value, median value, and peak mean value. The reliability is calculated based on the difference between the previous value of the distance to the light source or glossy surface and the distance corresponding to each of the mean value, median value, and peak mean value. The one or more processors set a reference value in the disparity distribution from among the mean value, median value, and peak mean value based on the reliability. Furthermore, the one or more processors calculate the distance to the light source or glossy surface based on a calculation region. The calculation region has a predetermined sampling width from the reference value. The width of the calculation region is less than the distance between the peaks at both ends.

[0014] According to the above configuration, the validity of the mean, median, and peak mean values ​​in the disparity distribution is determined based on the previous values. For example, if the mean value indicates a distance that is significantly different from the previous value of the distance between the vehicle and the light source, the reliability of the mean value becomes low.

[0015] In the above configuration, the one or more processors may set the value with the highest reliability among the average value, the median value, and the peak average value as the reference value.

[0016] According to the above configuration, the value that is least different from the previous value is set as the reference value.

[0017] In the above configuration, the one or more processors may set a value among the average, median, and peak average that exceeds a threshold for reliability as a reference value. In this case, when multiple reference values ​​are set, the one or more processors calculate a calculation area for each reference value. Furthermore, the one or more processors calculate the distance to the light source or glossy surface based on a congruent area where the multiple calculation areas are overlapped.

[0018] According to the above configuration, the union of the calculation areas based on the highly reliable reference values ​​is obtained as the final calculation area.

[0019] In the above configuration, the image processing device may further include a second stereo camera. The field of view of the second stereo camera overlaps with that of the stereo camera. The exposure time of the second stereo camera is shorter than that of the stereo camera. The one or more processors generate a difference image between a high-brightness area in an image captured by the stereo camera and a high-brightness area in an image captured by the second stereo camera. The one or more processors then adjust the exposure time of the stereo camera based on the brightness distribution in the difference image.

[0020] By shortening the exposure time, the striations on the captured image can be shortened. With the above configuration, the image area of ​​the striations is extracted from the high-brightness area using the differential image. The exposure time of the stereo camera is adjusted based on the size and length of the extracted striations. [Effects of the Invention]

[0021] According to the image processing device disclosed in this specification, even when a streak appears in a captured image, it is possible to suppress a decrease in accuracy in calculating the distance to the subject. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram illustrating a stereo camera and headlamps mounted on a vehicle. [Figure 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 3] 10A and 10B are diagrams illustrating examples of images captured by a right camera of a stereo camera. [Figure 4] 10A and 10B are diagrams illustrating examples of images captured by the left camera of the stereo camera. [Figure 5] 10A and 10B are diagrams illustrating a process of specifying a light source region after a captured image is binarized; [Figure 6] FIG. 1 is a diagram illustrating the principle of the generation of striations. [Figure 7] 10A and 10B are diagrams illustrating variations in parallax due to the occurrence of striations. [Figure 8] 10A and 10B are diagrams illustrating an example of a parallax distribution when a light striation appears in a light source area. [Figure 9] FIG. 10 is a diagram illustrating a flowchart of headlamp orientation control according to the present embodiment. [Figure 10] FIG. 10 is a diagram illustrating a flowchart of a reference value specification process. [Figure 11] FIG. 10 is a diagram illustrating an example of feature values ​​in a disparity distribution. [Figure 12] FIG. 10 is a diagram illustrating a reliability function. [Figure 13] FIG. 10 is a diagram illustrating a flowchart of a distance calculation process. [Figure 14] FIG. 10 is a diagram illustrating an example of a sampling width in a disparity distribution. [Figure 15] FIG. 10 is a diagram illustrating a flowchart according to a first modified example of the reference value specification process. [Figure 16] FIG. 10 is a diagram illustrating a flowchart according to a first modification of the distance calculation process. [Figure 17] 10A and 10B are diagrams illustrating a sampling width according to a first modification of the distance calculation process; [Figure 18] FIG. 10 is a diagram showing another example of an image processing device equipped with a two-set stereo camera system. [Figure 19] FIG. 10 is a diagram illustrating a process of generating a difference image. [Figure 20] FIG. 10 is a diagram illustrating a flowchart according to a first modification of the headlamp orientation control. [Figure 21] FIG. 10 is a flowchart illustrating a second modification of the reference value specification process. [Figure 22] FIG. 10 is a flowchart illustrating a second modification of the distance calculation process. DETAILED DESCRIPTION OF THE INVENTION

[0023] 1. Vehicle configuration Fig. 1 illustrates a vehicle 100. Fig. 2 illustrates a hardware configuration of an image processing device according to this embodiment. The image processing device according to this embodiment is mounted on the vehicle 100. This image processing device includes a stereo camera 20 and an AFS-ECU 10 (see Fig. 2).

[0024] The image processing device according to this embodiment is incorporated into, for example, a headlamp orientation control system. For example, the headlamp orientation control system is an automatic high beam control system. The headlamp orientation control system automatically switches the headlamps 31R, 31L between high beam and low beam. When switching the beam, the situation ahead of the vehicle 100 is reflected. For example, on a left-hand traffic road, when an oncoming vehicle is detected, the AFS-ECU 10 switches the right headlamp 31R from high beam to low beam.

[0025] The image processing device according to this embodiment calculates the distance between the headlights of oncoming vehicles and the taillights of preceding vehicles, particularly at night. When calculating the distance between the vehicle and a light source such as a headlight, light streaks 53RL and 53RR, as shown in FIG. 3, become noise components. Therefore, the image processing device performs processing to reduce the influence of light streaks 53RL and 53RR in the process of calculating the distance between the vehicle and the light source.

[0026] 1, a stereo camera 20 is mounted on, for example, the roof of a vehicle 100. A roof bar 30 is installed on a roof panel of the vehicle 100. The roof bar 30 extends in the width direction of the vehicle 100.

[0027] The stereo camera 20 is installed on the roof bar 30. The stereo camera 20 includes a right camera 21R and a left camera 21L. The fields of view of the right camera 21R and the left camera 21L at least partially overlap. The right camera 21R and the left camera 21L are positioned at the same height.

[0028] The horizontal (vehicle width) separation distance between the right camera 21R and the left camera 21L is also called the baseline. The baseline value is used as a parameter for distance calculation. Because the right camera 21R and the left camera 21L are fixed to the roof bar 30, the baseline value is stored in the AFS-ECU 10 (see FIG. 2) as a known fixed value.

[0029] In addition to the baseline and the parallax (described later), the focal lengths of the right camera 21R and the left camera 21L are also used in calculating the distance. The focal lengths of the right camera 21R and the left camera 21L are known from specifications and the like, and are stored in the AFS-ECU 10 (see FIG. 2).

[0030] FIG. 3 illustrates an image 50R captured by the right camera 21R. FIG. 4 illustrates an image 50L captured by the left camera 21L. Typically, as shown in the position of an oncoming vehicle 51L, there is a discrepancy in the position of the subject between the captured image 50R and the captured image 50L. This discrepancy is called parallax. The farther the subject is from the vehicle, the smaller the parallax. The closer the subject is from the vehicle, the larger the parallax.

[0031] Known image processing such as segmentation and annotation is used to identify objects in a captured image. Then, the distance between each object and the vehicle is calculated by calculating the parallax of each object. More specifically, the distance from the vehicle to the object can be calculated using the focal lengths, baselines, and parallax of the right camera 21R and the left camera 21L based on the principle of triangulation. This calculation process is well known, so a detailed description will be omitted below.

[0032] The AFS-ECU 10 also detects vehicle lighting at night or in dark places such as parking lots. By detecting vehicle lighting, it becomes possible to accurately detect the distance between the vehicle and other vehicles even in dark places. Details of the lighting detection process will be described later.

[0033] The right camera 21R and the left camera 21L are also equipped with an aperture mechanism. Referring to Fig. 6, the aperture mechanism includes a plurality of aperture blades 70A-70F. By narrowing the field of view to a certain extent using the aperture blade mechanism, a high-contrast captured image can be obtained. Obtaining a high-contrast captured image allows for accurate object and light source recognition within the captured image.

[0034] On the other hand, as will be described later, narrowing the field of view using the aperture blades 70A-70F causes light streaks 53RR and 53RL to appear in the captured image as illustrated in the lower part of Fig. 6. As will be described later, the light streaks 53RR and 53RL become noise components when calculating the distance between the light sources 52RR and 52RL and the host vehicle. As will be described later, the image processing device according to this embodiment calculates the distance between the light sources 52RR and 52RL and the host vehicle after performing processing to reduce the influence of the light streaks 53RR and 53RL.

[0035] 2 illustrates a plurality of devices constituting the headlamp orientation control system. The headlamp orientation control system includes a stereo camera 20, an adaptive front drive system ECU 10 (AFS-ECU), a right LED array 41R, and a left LED array 41L.

[0036] The right LED array 41R includes n LED elements 41R_1, 41R_2, ..., 41R_n. The LED elements 41R_1, 41R_2, ..., 41R_n are arranged, for example, in the vehicle width direction. The LED elements 41R_1, 41R_2, ..., 41R_n have their optical axes oriented in different directions, for example. For example, the odd-numbered LED elements 41R_2k+1, such as LED elements 41R_1, 41R_3, have their optical axes set as high beams. Furthermore, the even-numbered LED elements 41R_2k, such as LED elements 41R_2, 41R_4, have their optical axes set as low beams.

[0037] The left LED array 41L has a structure that is line-symmetrical to the right LED array 41R. For example, if the reference numeral "41R" in the above description is replaced with the reference numeral "41L," the description will be the same as that of the left LED array 41L.

[0038] The AFS-ECU 10 is an electronic control unit for the adaptive front lighting system. Part of the adaptive front lighting system performs headlamp orientation control, including the automatic high beam control described above.

[0039] The AFS-ECU 10 is a computing device including a CPU 11, a RAM 12, a ROM 13, a storage 14, a GPU 15, a VRAM 16, and an input / output controller 17.

[0040] The CPU 11 is a central processing unit, also called a processor. The RAM 12 is a volatile storage device that temporarily stores data during operation. The ROM 13 is a storage device from which data can be read. The storage 14 is a storage device from which data can be written and read. The storage 14 is configured, for example, by an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0041] The GPU 15 is an image processing device and is included in the processor. That is, the AFS-ECU 10 includes, as processors, at least the CPU 11 and the GPU 15. The VRAM 16 is a storage device dedicated to image processing.

[0042] The CPU 11 and the GPU 15 execute a program stored in the storage 14 or the ROM 13, which enables the CPU 11 and the GPU 15 to execute a headlamp orientation control flow exemplified in Fig. 9. In this control flow, the CPU 11 and the GPU 15 perform image processing on a pair of images captured by the stereo camera 20. Details of this control flow will be described later.

[0043] 2. Rays of light 6, the fields of view of the right camera 21R and the left camera 21L are narrowed by the aperture blades 70A-70F, so that in the captured image, light rays 53RR, 53RL, 53LR, and 53LL appear on 50R (see FIG. 3) and 50L (see FIG. 4).

[0044] The rays of light 53RR, 53RL, 53LR, and 53LL are streaks of light that extend radially from the light sources 52RR, 52RL, 52LR, and 52LL. ​​The rays of light 53RR, 53RL, 53LR, and 53LL are diffracted light that wraps around the rear surfaces of the diaphragm blades 70A-70F.

[0045] Note that streaks of light emitted from a light source are sometimes called "rays of light." Alternatively, streaks of light with a lower contrast ratio than striations are sometimes called "rays of light." Both rays of light and striations have in common the fact that they are streaks of light. Furthermore, in the image processing process according to this embodiment, the distance to the light source is calculated using the characteristics of the streaks of light extending from the light source. For this reason, streaks of light will be treated below as being included in striations of light.

[0046] 6, for example, a hexagonal aperture is formed by the aperture blades 70A-70F. The angle θ1 between the horizontal plane and the line connecting the vertex P1 closest to the horizontal plane and the optical axis P0 for this aperture is the arrangement angle of the rays 53RR and 53RL.

[0047] This arrangement angle θ1 is determined by how the diaphragm blades 70A-70F are arranged around the optical axis P0. The arrangement of the diaphragm blades 70A-70F with respect to the optical axis P0 varies depending on the camera. Therefore, the direction of the light striations, i.e., the arrangement angle θ1, differs between images captured by each camera.

[0048] 7 illustrates the parallax of the high-brightness region 65RL and the high-brightness region 65LL. The high-brightness region refers to an image region in which a light source and light striations extending from the light source are extracted within a captured image. The high-brightness region 65RL corresponds to the light source 52RL and light striations 53RL in the image 50R captured by the right camera 21R (see FIG. 3). More specifically, after the captured image 50R is subjected to a binarization process, the light source 52RL and light striations 53RL are extracted as the high-brightness region 65RL.

[0049] The high-brightness region 65LL corresponds to the light source 52LL and the light striations 53LL in the image 50L (see FIG. 4) captured by the left camera 21L. More specifically, after the captured image 50L is subjected to a binarization process, the light source 52LL and the light striations 53LL are extracted as the high-brightness region 65LL.

[0050] When the high-brightness regions 65RL and 65LL are superimposed, different parallaxes are calculated at multiple locations. For example, a parallax L1 is calculated between point P10 of the light source 52RL and corresponding point P12 of the light source 52LL. ​​Furthermore, for the rays 53RL and 53LL located below the light sources 52RL and 52LL, a parallax L2 is calculated between points P15 and P17 and their corresponding points P16 and P18. Furthermore, for the rays 53RL and 53LL located above the light sources 52RL and 52LL, a parallax L3 is calculated between points P20 and P22 and their corresponding points P21 and P23.

[0051] As shown by the parallaxes L1, L2, and L3, the parallax between pixels corresponding to a single light source varies based on the difference in the arrangement angles of the light rays 53RL and 53LL. Figure 8 shows an example of the parallax distribution for the high-luminance region 65RL and the high-luminance region 65LL. In the parallax distribution, the horizontal axis represents the parallax, and the vertical axis represents the frequency. For example, the parallax is calculated for all pixels in the high-luminance regions 65R and 65LL. The calculated parallax is statistically processed to form the parallax distribution.

[0052] As shown in Fig. 8, the disparity distribution has three major peaks F_peak1, F_peak2, and F_peak3. Based on the explanation of Fig. 7, the central peak F_peak2 is considered to indicate the "correct" disparity for the light sources 52RL and 52LL.

[0053] On the other hand, the peaks F_peak1 and F_peak3 at both ends are thought to represent "incorrect" disparities due to differences in the arrangement angles of the rays 53RL and 53LL. For example, the group with peak F_peak1 at its apex corresponds to the short disparity caused by the rays 53RL and 53LL, as shown in disparity L3. The group with peak F_peak3 at its apex corresponds to the long disparity caused by the rays 53RL and 53LL, as shown in disparity L2.

[0054] If the group with peak F_peak1 as its apex becomes a large group within the disparity distribution, a distance that is longer than the actual distance between the light sources 52RL, 52LL and the vehicle may be calculated. Similarly, if the group with peak F_peak3 as its apex becomes a large group within the disparity distribution, a distance that is shorter than the actual distance between the light sources 52RL, 52LL and the vehicle may be calculated.

[0055] Therefore, in the image processing device according to this embodiment, image processing capable of reducing the influence of striations in high-luminance areas is executed in the headlamp orientation control.

[0056] 3. Headlamp orientation control A control flow of the headlamp orientation control system is illustrated in Fig. 9. A program for executing the control flow of Fig. 9 is stored in the storage 14 or ROM 13 of the AFS-ECU 10 (see Fig. 2). The CPU 11 and GPU 15, which are processors, execute this program, enabling the CPU 11 and GPU 15 to execute each process of the control flow of Fig. 9.

[0057] The control flow in FIG. 9 is repeatedly executed at a predetermined frame rate. For example, the frame rate is set to 10 to 30 fps (frames per second). First, the stereo camera 20 captures an image of the area ahead of the vehicle (S10). Images 50R and 50L captured by the right camera 21R and the left camera 21L (see FIGS. 3 and 4) are sent to the AFS-ECU 10. Coordinates are assigned to each pixel of the captured images 50R and 50L based on a Cartesian coordinate system. For example, as illustrated in FIGS. 3 and 4, the upper left corners of the captured images 50R and 50L are the origins PR(0,0) and PL(0,0).

[0058] The CPU 11 performs binarization processing on the captured images 50R and 50L (S12). For example, in the captured images 50R and 50L, a value that is 70% of the maximum brightness is set as the threshold. Pixels with brightness below the threshold are corrected to a brightness of 0. Pixels with brightness exceeding the threshold are corrected to a brightness of 100.

[0059] FIG. 5 shows an excerpt of a binarized image 60R. The binarization process extracts the light sources 52RR and 52RL and the rays 53RR and 53RL. Note that the brightness of the rays 53RR and 53RL decreases the farther they are from the light sources 52RR and 52RL. The tips of the rays 53RR and 53RL are excluded from the binarized image in FIG. 5.

[0060] Next, the CPU 11 identifies high-luminance areas in the binarized image (S14). In identifying high-luminance areas, a group of pixels with a luminance of 100 is grouped. For example, the CPU 11 identifies a group of adjacent pixels with a luminance of 100 as a high-luminance area 65RL. The CPU 11 also identifies a group of pixels with a luminance of 100 that is located away from the high-luminance area 65RL as a high-luminance area 65RR. In the same manner, the CPU 11 identifies high-luminance areas in the binarized image of the captured image 50L.

[0061] A high-brightness region is an image region where a single light source and a light streak extending from the light source are extracted. For example, for a vehicle close to the host vehicle, the left and right headlights are spaced apart based on the resolution of the stereo camera 20, so multiple light sources are prevented from being included in a single high-brightness region.

[0062] Note that a process of recognizing a light source pair may be included between the process of identifying the high-brightness area (S14) and the process of stereo matching (S16). For example, if a pair of high-brightness areas is arranged at a predetermined interval in the horizontal direction, the CPU 11 recognizes that the pair of high-brightness areas is a vehicle lamp.

[0063] Since one high-luminance area corresponds to one light source, when calculating the disparity distribution of a high-luminance area, theoretically it is predicted that a unimodal distribution with a peak at a certain disparity will be obtained. In other words, when multimodality is recognized in the disparity distribution, it is thought that noise components are included in the high-luminance area.

[0064] Once the high-brightness areas have been determined in the binarized images of the captured images 50R and 50L, the GPU 15 performs stereo matching in each of the high-brightness areas (S16). As described above, a Cartesian coordinate system is assigned to the captured images 50R and 50L and their binarized images. Therefore, coordinates are assigned to pixels included in the high-brightness areas. The GPU 15 searches for pixels in one high-brightness area that correspond to pixels in the other high-brightness area. In other words, stereo matching associates pixels in one captured image with pixels in the other captured image. As a stereo matching method, for example, the well-known Sum of Absolute Difference (SAD) method is used.

[0065] Once the correspondence between pixels has been completed, the disparity, which is the distance between pixels, is calculated. The GPU 15 calculates the disparity for pixels in the high-precision region. The CPU 11 divides the disparity into multiple classes. The CPU 11 then counts the frequency of each class to create a disparity distribution (see FIG. 8) (S18).

[0066] Furthermore, the CPU 11 determines a frequency threshold value f_th for the disparity distribution. For example, the CPU 11 determines a value that is 15% of the maximum frequency F_peak2 in the disparity distribution as the frequency threshold value f_th.

[0067] The CPU 11 determines whether there are three or more peak values ​​exceeding the frequency threshold f_th (S20). In other words, the CPU 11 determines whether the disparity distribution is a multi-modal distribution having three or more peak values. Note that if the width of one section (class width) of the disparity distribution is narrow and the frequency fluctuates significantly, smoothing using a moving average is performed. Then, the number of peak values ​​is determined based on the disparity distribution after smoothing.

[0068] When the peak value exceeding the frequency threshold f_th in the disparity distribution is less than 3, the distance between the light source and the vehicle is calculated using all the disparity values ​​listed in the disparity distribution (S22). For example, the average value of the disparity in the disparity distribution is calculated. This average value becomes the representative value of the disparity.

[0069] In step S20, if there are three or more peak values ​​exceeding the frequency threshold f_th, the CPU 11 acquires the three largest peak values ​​(S24). As shown in the parallaxes L2 and L3 in the schematic diagram of FIG. 7 above, when multi-peaks are observed in the parallax distribution for a single light source, the light rays 53RL and 53LL are considered to be the main cause of the multi-peaks. Therefore, a process is performed to separate the two peaks of "incorrect" parallax corresponding to the light rays from the peak of "correct" parallax corresponding to the light source. In other words, the peak values ​​at both ends of the parallax distribution are excluded by the subsequent process.

[0070] After acquiring the peak values ​​of the three points, the CPU 11 performs a reference value determination process (S30). Referring to FIG. 14, the average value within the calculation region is calculated through the processes of steps S30 and S40. This average value becomes the representative value of the disparity. Referring to FIG. 14, the reference value V1 is a value that serves as a reference for determining the calculation region W3. The reference value determination process will be described in detail later.

[0071] The reference value V1 is calculated by the reference value calculation process. Furthermore, the CPU 11 executes a distance calculation process (S40). The distance calculation process determines the sampling widths W1 and W2 in FIG. 14. Furthermore, a calculation area W3 is determined by connecting the sampling widths W1 and W2. By identifying the calculation area W3, a representative value of the parallax is determined. The distance calculation process will be described in detail later.

[0072] In step S40 or step S22, the distance from the host vehicle to the light sources 52RR and 52RL is calculated. Furthermore, the GPU 15 determines the vehicle type of the oncoming vehicle 51R equipped with the light sources 52RR and 52RL from the captured image 50R (see FIG. 3) or the captured image 50L (see FIG. 4) before binarization processing (S26). For example, a tall SUV or truck does not need to switch from high beam to low beam even if it is close to the host vehicle. Based on the distance from the host vehicle to the light sources 52RR and 52RL and the vehicle type, the CPU 11 sets the high beam / low beam for the LED arrays 41R and 41L (see FIG. 2) (S28).

[0073] 4. Reference value identification process 10 illustrates an example of the flow of the reference value determination process. The CPU 11 calculates the relative speed between the vehicle and a light source (e.g., light sources 52LL and 52RL) corresponding to the parallax distribution (S31). For example, the relative speed of the light source can be obtained using a so-called tracking technique. The CPU 11 then calculates a current estimate of the distance between the vehicle and the light source from the relative speed, the previous distance value, and the frame rate (S32). The current estimate is not calculated directly from distance measurement by the stereo camera 20 (see FIG. 1), but is calculated based on the previous value.

[0074] Furthermore, the CPU 11 creates a reliability filter based on the current estimated value (S33). An example of the reliability filter is shown in FIG. 12. The reliability filter is a probability distribution. For example, the reliability filter has a normal distribution with the current estimated value as its peak. The horizontal axis indicates the difference between the distance based on the target value and the current estimated value. Details of the target value will be described later. The vertical axis indicates the probability (reliability). The reliability is expressed in %. A reliability threshold R_th is also set in the reliability filter. For example, the reliability threshold R_th is set to 50%.

[0075] When setting sampling widths W1 and W2 (see FIG. 14), the maximum reliability may be less than 100% to ensure that the peak values ​​F_peak1 and F_peak3 are excluded. For example, the maximum reliability is set to 80%.

[0076] 11, the CPU 11 calculates the median F_med, the average F_ave, and the three-peak average F_3peak in the disparity distribution as target values ​​(S34). The median F_med and the average F_ave are calculated from all values ​​in the disparity distribution. The three-peak average F_ave is the average of the three peak values ​​F_peak1, F_peak2, and F_peak3 in the disparity distribution.

[0077] The CPU 11 calculates the distance when the median value F_med, the average value F_ave, and the 3-peak average value F_3peak are used as disparities. The CPU 11 then calculates the difference between each of the calculated distances and the current estimated value.

[0078] Furthermore, the CPU 11 plots each of the calculated differences on a reliability filter (FIG. 12). Then, of these three values, the CPU 11 excludes values ​​that are less than the reliability threshold R_th from the candidates for the reference value V1 (S35). For example, if the difference value based on the average value F_ave is less than the reliability threshold R_th, the CPU 11 excludes the average value F_ave from the candidates for the reference value V1.

[0079] The current estimated value is the estimated value of the distance between the vehicle and the light source at the current time. Values ​​that deviate significantly from this estimated value are excluded in step S35.

[0080] After applying the reliability filter, the CPU 11 determines whether any values ​​remain as candidates for the reference value V1 (S36). If there are no candidates, the CPU 11 determines that there is no reference value (S37). On the other hand, if there are remaining candidates for the reference value, a reference value is selected from the remaining candidates based on their reliability. For example, the CPU 11 sets the value with the highest reliability among the remaining candidates as the reference value V1 (S38).

[0081] 5. Distance calculation process 13, the CPU 11 determines whether or not the reference value V1 has been calculated in the reference value determination process (S41). If the reference value V1 has not been calculated, the CPU 11 outputs the current estimated value as the distance between the vehicle and the light source (S42).

[0082] In step S41, if the reference value V1 has been calculated, the CPU 11 determines the value obtained by multiplying the width from the reference value V1 to the left peak value F_peak1 by the reliability of the reference value as the sampling width W2 (S43).

[0083] Next, the CPU 11 multiplies the width from the reference value V1 to the right peak value F_peak3 by the reliability of the reference value, and sets the resulting value as the sampling width W1 (S44).

[0084] As described above, the reliability is a value less than 100%. Therefore, the sampling widths W1 and W2 are less than the inter-peak distances (the distance between F_peak1 and F_peak2 and the distance between F_peak2 and F_peak3).

[0085] Next, the CPU 11 determines a calculation region W3 (S45). Specifically, referring to FIG. 14, the width from the reference value V1 to the left (0 side) is determined as a sampling width W2. Also, the width from the reference value V1 to the right is determined as a sampling width W1. The sum of the sampling widths W1 and W2 is the calculation region W3. The calculation region W3 is less than the distance between the peaks at both ends of the disparity distribution (the distance between F_peak1 and F_peak3).

[0086] The CPU 11 calculates the distance between the light source and the vehicle based on the parallax in the calculation area (S46). The CPU 11 calculates the average value of the parallax in the calculation area W3. This average value becomes the representative value of the parallax.

[0087] 6. First Alternative Example of Reference Value Calculation Processing and Distance Calculation Processing Fig. 15 shows a first example of the reference value calculation process. Steps with the same reference numerals as in Fig. 10 have the same processing content as in Fig. 10. Therefore, in the following, explanations of steps with the same reference numerals as in Fig. 10 will be omitted as appropriate.

[0088] In step S36, the CPU 11 determines whether or not there are any remaining values ​​as candidates for the reference value V1. If there are any remaining candidates for the reference value, the CPU 11 sets each of the remaining values ​​as the reference value (S138). Note that the following describes an example in which multiple reference values ​​V1 and V2 are set.

[0089] 16 shows another example of distance calculation processing. In step S41, if there are multiple reference values ​​V1 and V2, CPU 11 sets reference value count k to an initial value of 1 (S140). Then, through the processing of steps S43 to S45, CPU 11 determines sampling widths W1_1 (see FIG. 17) and W2_1 for reference value V1. In other words, CPU 11 determines calculation regions for each reference value.

[0090] Furthermore, the CPU 11 determines whether the reference value count k is the final value (S141). If the reference value count k has not reached the final value, the CPU 11 increments the reference value count (S142). Then, the processes from steps S43 to S45 are executed.

[0091] When sampling widths W1 and W2 are determined for all reference values ​​V1 and V2, sampling widths W1_1 and W2_1 are determined for the reference value V1 in the disparity distribution, as illustrated in Fig. 17. Sampling widths W1_2 and W2_2 are determined for the reference value V2.

[0092] The CPU 11 determines the union of these sampling widths as the calculation area W3 (S143). In other words, the joint area where the calculation area for the reference value V1 and the calculation area for the reference value V2 are overlapped becomes the final calculation area.

[0093] In other words, the larger of the value separated by the sampling width W1_1 from the reference value V1 and the value separated by the sampling width W1_2 from the reference value V2 is the maximum value of the calculation area. Similarly, the smaller of the value separated by the sampling width W2_1 from the reference value V1 and the value separated by the sampling width W2_2 from the reference value V2 is the minimum value of the calculation area.

[0094] 7.2-set stereo camera system 18 shows another example of a headlamp orientation control system. This system is exemplified by a two-set stereo camera system. That is, the headlamp orientation control system includes a first stereo camera 20, a second stereo camera 25, an adaptive front drive system ECU 10 (AFS-ECU), a right LED array 41R, and a left LED array 41L. Since the configuration other than the second stereo camera 25 has already been explained, explanations thereof will be omitted below as appropriate.

[0095] The second stereo camera 25 includes a right camera 26R and a left camera 26L. The right camera 26R and the left camera 26L are fixed to the roof bar 30. For example, the distance (base line) between the right camera 26R and the left camera 26L is shorter than the baseline between the right camera 21R and the left camera 21L of the first stereo camera 20. Furthermore, the fields of view of the first stereo camera 20 and the second stereo camera 25 overlap at least partially.

[0096] The right camera 26R and the left camera 26L are set to have shorter exposure times than the right camera 21R and the left camera 21L. If the exposure time is short, the captured image will be dark, making object recognition difficult. On the other hand, if the exposure time is short, the size of the light stripes will be small. The exposure time of the first stereo camera 20 is adjusted using the second stereo camera 25.

[0097] 19, an image captured by the right camera 21R of the first stereo camera 20 is transmitted to the AFS-ECU 10. An image captured by the right camera 26R of the second stereo camera 25 is also transmitted to the AFS-ECU 10.

[0098] The CPU 11 of the AFS-ECU 10 binarizes each of the pair of captured images. Furthermore, the CPU 11 identifies high-brightness areas. Furthermore, the CPU 11 generates a difference image 95 by superimposing the high-brightness areas of one of the binarized images 60R (see FIG. 19) and the high-brightness areas of the binarized image 90R. For example, the CPU 11 calculates the geometric centers of the high-brightness areas in the binarized images 50R and 90R. The high-brightness areas are then superimposed so that their geometric centers coincide.

[0099] Because the second stereo camera 25 has a short exposure, substantially no striations appear in the binarized image 90R. Therefore, only the striations 53RR and 53RL are extracted in the difference image 95. The CPU 11 adjusts the exposure time of the first stereo camera 20 based on the luminance distribution in the difference image. For example, the CPU 11 adjusts the exposure time of the first stereo camera 20 based on the size of the striations 53RR and 53RL in the difference image 95. For example, if the total area of ​​the striations 53RR and 53RL is 50% or more of the total area of ​​the light sources 52RR and 52RL, the CPU 11 shortens the exposure time of the first stereo camera 20 by a predetermined percentage.

[0100] 8. First Variant of Headlamp Orientation Control and Second Variant of Reference Value Identification Processing and Distance Calculation Processing Fig. 20 shows a flowchart of a first modified example of headlamp orientation control, Fig. 21 shows a flowchart of a second modified example of reference value determination processing, and Fig. 22 shows a flowchart of a second modified example of distance calculation processing.

[0101] 20, 21, and 22, steps having the same reference numerals as those in Figures 9, 10, and 13 have the same processing contents as those in Figures 9, 10, and 13. Therefore, in the following, explanations of steps having the same reference numerals as those in Figures 9, 10, and 13 will be omitted as appropriate.

[0102] The disparity distribution in Fig. 8 shows three major peaks: F_peak1, F_peak2, and F_peak3. In the above-described embodiment, when calculating the distance to the light source, the peaks F_peak1 and F_peak3 at both ends, which are "false" peaks, are excluded. In the embodiments illustrated in Figs. 20, 21, and 22, the distance between the light source and the host vehicle is calculated based on a disparity distribution having two or more peaks.

[0103] In step S18, when the disparity distribution of the high-luminance region (see FIG. 8) is obtained, the CPU 11 determines whether there are two or more peak values ​​exceeding the frequency threshold f_th (S230). In other words, the CPU 11 determines whether the disparity distribution is a multi-peak distribution having two or more peak values. Note that if the width of one section (class width) of the disparity distribution is narrow and the frequency fluctuates significantly, smoothing using a moving average is performed. Then, the number of peak values ​​is determined based on the disparity distribution after smoothing.

[0104] When there are two or more peak values ​​exceeding the frequency threshold f_th, the CPU 11 executes a reference value specification process (S230) and a distance calculation process (S240). Fig. 21 illustrates a second example of the reference value specification process.

[0105] The CPU 11 calculates the median F_med, the average F_ave, and the peak average F_peaks in the disparity distribution (S234). The peak average F_peaks is the average of a plurality of peak values ​​that exceed the threshold frequency.

[0106] The CPU 11 calculates the distance when the median value F_med, the average value F_ave, and the peak average value F_peaks are used as the disparity. The CPU 11 then calculates the difference between each of the calculated distances and the current estimated value.

[0107] Furthermore, the CPU 11 plots each of the calculated differences on the reliability filter (FIG. 12).Then, of these three values, the CPU 11 excludes values ​​that are less than the reliability threshold R_th from candidates for the reference value V1 (S235).

[0108] 22, when the reference value V1 is calculated in step S41, the CPU 11 determines the sampling width W2 as a value obtained by multiplying the width from the reference value V1 to the minimum value of the disparity distribution by the reliability of the reference value (S243). Here, the minimum value of the disparity distribution may be extracted from values ​​exceeding a threshold frequency.

[0109] Next, the CPU 11 multiplies the width from the reference value V1 to the maximum value of the disparity distribution by the reliability of the reference value V1 to determine the sampling width W1 (S244). Here, the maximum value of the disparity distribution may be extracted from values ​​exceeding a threshold frequency.

[0110] Next, the CPU 11 calculates a calculation region W3 (S45). Specifically, as in FIG. 14, the width on the left side (0 side) of the reference value V1 is determined as the sampling width W2. Also, the width on the right side of the reference value V1 is determined as the sampling width W1. With the above configuration, when multiple peaks appear in the disparity distribution, disparities that are significantly different from the reference value are excluded.

[0111] 9. Other Embodiments In the above-described embodiment, the image processing device according to this embodiment is incorporated into a part of a headlamp orientation control system. However, the image processing device according to this embodiment is not limited to this embodiment. For example, the stereo camera 20 is installed on the vehicle 100 so that the field of view is directed to the side or rear of the vehicle. Then, particularly at night or in a parking lot, the image processing device determines the positions of the vehicle and vehicles approaching from the side or rear of the vehicle 100.

[0112] In the image processing process described above, the process of suppressing the effects of streaks caused by light sources has been described. Here, in addition to light sources, which are luminous bodies, glossy surfaces are also included as targets for suppressing the effects of streaks. That is, in addition to light sources, glossy surfaces are also extracted as high-brightness areas in the target of the headlamp orientation control flowchart shown in FIG. 9.

[0113] When strong light is incident on a shiny surface such as a puddle on the road, streaks of light appear from the shiny surface in the captured image. At this time, the shiny surface and the streaks of light extending from the shiny surface are extracted as a high-brightness area. For this high-brightness area, the processing from step S24 onward is executed in the headlamp orientation control of FIG. 9. That is, the CPU 11 executes processing to suppress the influence of streaks of light on the shiny surface.

[0114] In the above embodiment, image processing is performed by two types of processors, the CPU 11 and the GPU 15. Alternatively, the above image processing may be performed by the CPU 11 alone. [Explanation of symbols]

[0115] 10 AFS-ECU, 11 CPU (processor), 20 stereo camera (first stereo camera), 21L left camera of first stereo camera, 21R right camera of first stereo camera, 25 second stereo camera, 26L left camera of second stereo camera, 26R right camera of second stereo camera, 31L headlamp, 52LL, 52LR, 52RL, 52RR light source, 53LL, 53LR, 53RL, 53RR light stripe, 65LL, 65R, 65RL, 65RR high brightness area, 70A-70F aperture blade, 95 difference image, 100 vehicle.

Claims

1. A stereo camera mounted on the vehicle to capture images outside the vehicle; one or more processors that perform image processing on the pair of images captured by the stereo camera; An image processing device comprising: One or more of the processors Identifying a high-brightness region, which is an image region from which a single light source or a shiny surface and a light striation extending from the light source or the shiny surface are extracted, from each of the captured images; Corresponding pixels to each other for each of the high-luminance regions; determining a parallax distribution in the high-luminance region based on the correspondence; calculating a distance to the light source or the glossy surface based on the parallax distribution; Image processing device.

2. 2. The image processing device according to claim 1, One or more of the processors When the disparity distribution is a multi-modal distribution having three or more peak values, excluding the peak values ​​at both ends of the disparity distribution, calculating a distance to the light source or the glossy surface based on the parallax distribution after the exclusion; Image processing device.

3. 3. The image processing device according to claim 2, One or more of the processors calculating a mean value, a median value, and a peak mean value, which is an average value of a plurality of the peak values, in the disparity distribution before excluding the peak values ​​at the both ends; calculating reliability for the average value, the median value, and the peak average value based on differences between a previous value of the distance from the light source or the glossy surface and distances corresponding to the average value, the median value, and the peak average value, respectively; setting a reference value in the disparity distribution from among the average value, the median value, and the peak average value based on the reliability; calculating a distance from the light source or the glossy surface based on a calculation area having a predetermined sampling width from the reference value; The width of the calculation region is less than the distance between the peaks at both ends. Image processing device.

4. 4. The image processing device according to claim 3, One or more of the processors the value having the greatest reliability among the average value, the median value, and the peak average value is set as the reference value; Image processing device.

5. 4. The image processing device according to claim 3, One or more of the processors setting a value among the average value, the median value, and the peak average value that exceeds a threshold value for the reliability as the reference value; When a plurality of reference values ​​are set, the calculation region is calculated for each of the reference values; calculating a distance to the light source or the glossy surface based on a joint area in which the plurality of calculation areas are overlapped; Image processing device.

6. 3. The image processing device according to claim 1, a second stereo camera whose field of view overlaps with that of the stereo camera; an exposure time of the second stereo camera is shorter than an exposure time of the stereo camera; One or more of the processors generating a difference image between the high-brightness area in the image captured by the stereo camera and the high-brightness area in the image captured by the second stereo camera; adjusting an exposure time of the stereo camera based on the luminance distribution in the difference image; Image processing device.

Images