Image processing device and image processing method

Abstract

The present invention provides an image processing device which, even in the case of flare occurring at photographing of a stereoscopic object provided with a high luminance light source, can accurately calculate disparity information on the surroundings of the high luminance light source and can estimate, with accuracy, the position, the speed, the type, and the like of the stereoscopic object. The image processing device comprises: a camera that captures a first image with a first exposure value and captures a second image with a second exposure value smaller than the first exposure value; a stereoscopic object extraction unit that extracts, from the first image, a first region where a stereoscopic object is present and extracts, from the second image, a second region where the stereoscopic object is present; a stereoscopic object information detection unit that detects first stereoscopic object information from the first region and detects second stereoscopic object information from the second region; and a stereoscopic object information integration unit that integrates the first stereoscopic object information and the second stereoscopic object information.

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method for recognizing an environment outside a vehicle based on an image captured outside the vehicle. [Background technology]

[0002] In recent years, automobiles equipped with driving assistance functions that assist with acceleration / deceleration and steering operations according to the outside environment are becoming more common, and the need for driving assistance functions during night driving is also increasing. In order to provide appropriate driving assistance during night driving, it is necessary to accurately estimate the distance to the preceding vehicle and the speed of the preceding vehicle regardless of the brightness of the brake lights of the preceding vehicle. For this reason, recent image processing devices are required to have improved functions for accurately detecting light sources from low to high brightness, and technologies such as HDR (High Dynamic Range) that expand the dynamic range during imaging are being used.

[0003] However, when the dynamic range during imaging is widened, a flare (halation) phenomenon that makes the surroundings of a high-brightness light source (for example, a brake light during braking) brightly and blurry is likely to occur when imaging the light source. When flare occurs, there is a problem that three-dimensional objects cannot be detected correctly in the flare area.

[0004] Therefore, the image processing device in the abstract of Patent Document 1 "provides an image processing device capable of capturing an image of a high-luminance target without causing halation using a single camera even at night, etc.", "equipped with a camera and a control unit mounted on a vehicle to capture an image of the vehicle's surroundings, the control unit extracts the vehicle from the image captured by the camera, and calculates an area for correcting brightness in the extracted vehicle and its vicinity based on the brightness in the image of the vehicle's surroundings captured by the camera. Then, in the calculated brightness correction area, the parts with brightness that cause halation, such as the headlight part, the headlight road reflection part, and the headlight flare part, are calculated, and the calculated parts that cause halation are corrected to a brightness that does not cause halation by multiplying them by a specified reduction rate, and the corrected halation area is output by being superimposed on the image captured by the camera." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-073009 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the image processing device of Patent Document 1, brightness correction is performed only on the area causing halation, so that the brightness change is discontinuous inside and outside the boundary of the correction area. As a result, it is not possible to accurately calculate disparity information (distance information for each point on the image) near the boundary, and there is a problem that the distance and speed of objects near the boundary (such as a preceding vehicle) cannot be accurately estimated.

[0007] Therefore, an object of the present invention is to provide an image processing device that can accurately calculate disparity information around a high-brightness light source even if flare occurs when imaging a three-dimensional object equipped with a high-brightness light source, and can accurately estimate the position, speed, shape, type, etc. of the three-dimensional object. [Means for solving the problem]

[0008] In order to solve the above problems, the image recognition device of the present invention is an image processing device that includes a camera that captures a first image with a first exposure amount and captures a second image with a second exposure amount that is less than the first exposure amount, a three-dimensional object extraction unit that extracts a first area in which a three-dimensional object exists from the first image and extracts a second area in which the three-dimensional object exists from the second image, a three-dimensional object information detection unit that detects first three-dimensional object information from the first area and detects second three-dimensional object information from the second area, and a three-dimensional object information integration unit that integrates the first three-dimensional object information and the second three-dimensional object information. Effect of the Invention

[0009] According to the image processing device of the present invention, even if flare occurs when imaging a three-dimensional object having a high-brightness light source, it is possible to accurately calculate disparity information around the high-brightness light source, and to accurately estimate the position, speed, shape, type, etc. of the three-dimensional object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a hardware configuration diagram of the stereo camera device according to the embodiment. [Diagram 2] FIG. 1 is a functional block diagram of a stereo camera device according to an embodiment. [Diagram 3] 4 is a process flowchart of the stereo camera device according to the embodiment. [Figure 4] 13 is a flowchart of a three-dimensional object recognition process according to a comparative example. [Figure 5A] 4 is a flowchart of a three-dimensional object recognition process according to the present invention. [Figure 5B] 13 shows an example of a procedure for calculating the amount of flare light. [Figure 6A] FIG. 4 is a functional block diagram showing details of an object recognition unit. [Figure 6B] FIG. 4 is a functional block diagram showing details of a normal shutter three-dimensional object processing unit. [Figure 6C] FIG. 4 is a functional block diagram showing details of a low exposure shutter three-dimensional object processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a stereo camera device, which is an embodiment of an image processing device of the present invention, will be described with reference to the drawings.

[0012] <Hardware configuration diagram of the stereo camera device 10> FIG. 1 is a hardware configuration diagram showing an outline of a stereo camera device 10 according to the present embodiment. The stereo camera device 10 is an in-vehicle device that recognizes the vehicle exterior environment based on captured images of the vehicle exterior, such as white lines on the road, pedestrians, other vehicles, other three-dimensional objects, traffic signals, signs, lit lamps, etc. Then, the stereo camera device 10 determines a control policy such as acceleration / deceleration support and steering support for the vehicle according to the recognized vehicle exterior environment.

[0013] 1, the stereo camera device 10 includes cameras 11 (left camera 11L, right camera 11R), an image input interface 12, an image processing unit 13, an arithmetic processing unit 14, a storage unit 15, a CAN interface 16, and an abnormality monitoring unit 17. The components from the image input interface 12 to the abnormality monitoring unit 17 are configured as a single or multiple computer units interconnected via an internal bus, and various functions of the image processing unit 13, the arithmetic processing unit 14, etc. are realized by an arithmetic device (CPU, etc.) included in the computer unit executing a predetermined program, but the present invention will be described below while omitting such well-known techniques.

[0014] The camera 11 is a stereo camera that is installed on the vehicle so as to capture an image of the area in front of the vehicle, and is composed of a left camera 11L and a right camera 11R. The imaging element of the left camera 11L captures a first left image P1 with a normal exposure (hereinafter, “normal shutter”). L and a second left image P2 with a lower exposure than normal (hereinafter referred to as "low exposure shutter"). L In parallel with the image capture by the left camera 11L, the image capture element of the right camera 11R captures a first right image P1 R and the second right image P2 with a low exposure shutter R are alternately captured.

[0015] Here, the "exposure amount" in this embodiment is a physical amount obtained by multiplying the exposure time of the image sensor by a gain for amplifying the output of the image sensor. The exposure amount of the "normal shutter" is, for example, an exposure time of 20 ms multiplied by a gain 16 times the low-exposure shutter gain G described later, and the exposure amount of the "low-exposure shutter" is, for example, an exposure time of 0.1 ms multiplied by a predetermined low-exposure shutter gain G, but each exposure amount is not limited to the above examples.

[0016] In the above, the first image P1 (P1 L , P1 R ) and the second image P2 (P2 L , P2 R However, under certain conditions, the capture of the second image P2 with a low exposure shutter may be omitted. For example, when the luminance value of the first image P1 with a normal shutter or an illuminance sensor determines that the external light is dark, or when the lamp size W1 (see FIG. 5B) in the first image P1 described later is smaller than a predetermined threshold value W th Only when it is determined that this is the case, the second image P2 may be captured with a low exposure shutter.

[0017] The image input interface 12 controls the image capture by the camera 11 and outputs the captured image P (P1 L , P2 L , P1 R , P2 R ) is input via this interface. The image P input via this interface is sent to the image processing unit 13 and the like via the internal bus.

[0018] The image processor 13 processes the left image P captured by the left camera 11L. L (P1 L , P2 L ) and the right image P captured by the right camera 11R R (P1 R , P2 R ) are compared, and each image is subjected to correction such as correction for device-specific deviations caused by the imaging element and noise interpolation, and then the corrected image P is stored in the storage unit 15.

[0019] The image processor 13 also calculates corresponding points between the left and right images with the same exposure, calculates parallax information I (distance information for each point on the image), and stores the information in the memory 15. Specifically, the first left image P1 L and the first right image P1 R The first parallax information I1 is calculated from the comparison of the second left image P2 captured at the same time. L and the second right image P2 R The second parallax information I2 is calculated from the comparison.

[0020] The arithmetic processing unit 14 uses the image P and the parallax information I stored in the memory unit 15 to recognize various objects necessary for perceiving the environment around the vehicle. The various objects recognized here include people, other vehicles, other obstacles, traffic lights, signs, tail lights and headlights of other vehicles, etc. These recognition results and some of the intermediate calculation results are recorded in the memory unit 15. Furthermore, the arithmetic processing unit 14 uses the object recognition results to determine a control policy for the host vehicle.

[0021] The memory unit 15 is a storage device such as a semiconductor memory, and stores the corrected image P and disparity information I that are outputs of the image processing unit 13, as well as the object recognition results and the vehicle control policy that are outputs of the calculation processing unit 14.

[0022] The CAN interface 16 is an interface that transmits the object recognition results obtained by the arithmetic processing unit 14 and the vehicle's control policy to an in-vehicle network CAN (Controller Area Network). In addition, a control system (ECU, etc.) that controls the vehicle's drive system, braking system, steering system, etc. is connected to the in-vehicle network CAN, and the stereo camera device 10 can perform driving assistance such as automatic braking and steering avoidance in accordance with the environment outside the vehicle by intervening the control system.

[0023] The abnormality monitoring unit 17 monitors whether each part in the stereo camera device 10 is operating abnormally, whether an error has occurred during data transfer, etc., and serves as a mechanism for preventing abnormal operation.

[0024] <Functional block diagram of stereo camera device 10> Next, a basic processing flow of the stereo camera device 10 will be described with reference to the functional block diagram of FIG.

[0025] As described above, the left camera 11L captures the first left image P1 L and the second left image P2 with low exposure shutter L The right camera 11R also captures the first right image P1 R and the second right image P2 with low exposure shutter R are alternately captured.

[0026] The image correction unit 13a of the image processing unit 13 corrects each image P (P1 L , P2 L , P1 R , P2 R ) is subjected to image correction processing that absorbs the inherent characteristics of the imaging element. Then, the corrected image P is stored in the image buffer 15a in the storage unit 15.

[0027] Thereafter, the parallax calculation unit 13b of the image processing unit 13 calculates the parallax between the left and right images captured at the same timing (first left image P1 L and the first right image P1 R , or the second left image P2 L and the second right image P2 R ) and calculate the parallax information I for each exposure. L Where on the top and the right image P R Since it is clear which of the above corresponds, the distance to the object on each image can be obtained by the principle of triangulation. The parallax information I (I1, I2) for each exposure amount obtained here is stored in the parallax buffer 15b of the storage unit 15.

[0028] Furthermore, the object recognition unit 14a of the calculation processing unit 14 performs object recognition processing to extract an area where a three-dimensional object exists, using the image P and the parallax information I stored in the storage unit 15. The three-dimensional objects to be recognized include, for example, people, cars, other three-dimensional objects, signs, traffic lights, tail lights, etc., and when recognizing, the recognition dictionary 15c is referred to as necessary.

[0029] After completing the object recognition, the vehicle control unit 14b of the calculation processing unit 14 determines a control policy for the vehicle by considering the object recognition result output from the object recognition unit 14a and the state of the vehicle (speed, steering angle, etc.). For example, if there is a possibility of a collision with a preceding vehicle, a control signal is generated to issue a warning to the occupants to urge them to operate to avoid the collision, or to control the vehicle to avoid the preceding vehicle by controlling the braking and steering angle, and is output to a control system (ECU, etc.) via the CAN interface 16 and the in-vehicle network CAN.

[0030] <Processing flow chart of image processing unit and calculation processing unit> Next, the processing performed by the image processing unit 13 and the calculation processing unit 14 will be described with reference to the flowchart of FIG.

[0031] First, in step S1, the image correction unit 13a of the image processing unit 13 corrects the right image P R Specifically, the image correction unit 13a processes the right image P captured by the right camera 11R. R (P1 R , P2 R ) is subjected to device-specific deviation correction, noise correction, and the like, and then stored in the image buffer 15 a of the storage unit 15 .

[0032] Next, in step S2, the image correction unit 13a of the image processing unit 13 corrects the left image P L Specifically, the image correction unit 13a processes the left image P captured by the left camera 11L. L (P1 L , P2 L ) is subjected to device-specific deviation correction, noise correction, and the like, and then stored in the image buffer 15a of the storage unit 15. Note that the order of steps S1 and S2 may be reversed.

[0033] Then, in step S3, the parallax calculation unit 13b of the image processing unit 13 calculates the corrected left image P L and the right image P R The parallax information I thus obtained is stored in the parallax buffer 15 b of the storage unit 15 .

[0034] Next, in step S4, the object recognition unit 14a of the arithmetic processing unit 14 recognizes the left image P L Or right image P R and the disparity information I. In this step, a three-dimensional object such as a preceding vehicle is also recognized, and the details of the three-dimensional object recognition method of this embodiment will be described later.

[0035] Finally, in step S5, the vehicle control unit 14b of the arithmetic processing unit 14 determines a vehicle control policy based on the object recognition result, and outputs the result to the in-vehicle network CAN.

[0036] <Comparative Example of Three-Dimensional Object Recognition Method> Here, the details of the three-dimensional object recognition method performed by the image processing device of the comparative example at the timing of step S4 in FIG. 3 will be described with reference to the flowchart of FIG. 4. Note that in the image processing device of the comparative example, the second image P2 (second left image P2 L , 2nd right image P2 R ), so in step S3 of FIG. 3, it is assumed that only the first parallax information I1 based on the first image P1 captured with the normal shutter is calculated.

[0037] First, in step S41, a three-dimensional object in the first image P1 is extracted based on the parallax distribution in the first parallax information I1 calculated in step S3 in Fig. 3, and the shape of the three-dimensional object and the distance to the three-dimensional object are estimated. In addition, a history of the distance to the three-dimensional object is recorded, and the speed of the three-dimensional object is estimated from the history information.

[0038] Next, in step S42, the recognition dictionary 15c is referenced to determine the type of the three-dimensional object detected in step S41.

[0039] Finally, in step S43, the three-dimensional object information (shape, distance, speed) estimated in step S41 and the three-dimensional object type determined in step S42 are output to the downstream vehicle control unit 14b.

[0040] The image processing device of the comparative example obtains three-dimensional object information (shape, distance, speed) and three-dimensional object type by the above method, but if flare occurs when capturing an image of a high-brightness light source, the parallax information I calculated in step S3 becomes inaccurate around the high-brightness light source, and the three-dimensional object information around the high-brightness light source detected in step S41 also becomes inaccurate. As a result, in the image processing device of the comparative example, the three-dimensional object type around the high-brightness light source determined in step S42 is also unreliable, and there is a possibility that driving assistance control based on unreliable information is inappropriate.

[0041] <Three-dimensional object recognition method of this embodiment> Next, the details of the three-dimensional object recognition method performed by the stereo camera device 10 of this embodiment at the timing of step S4 in FIG. 3 will be described with reference to the flowchart of FIG. 5A. In this embodiment, the first image P1 (P1 L , P1 R ), a second image P2 with a low exposure shutter (P2 L , P2 R ) are also captured, so in step S3 of FIG. 3, it is assumed that both the first parallax information I1 based on the first image P1 and the second parallax information I2 based on the second image P2 are calculated.

[0042] First, in step S41, the object recognition unit 14a detects information about a three-dimensional object based on each of the parallax distributions in the two types of parallax information I calculated in step S3 of FIG. 3. Specifically, the area of ​​the three-dimensional object in the first image P1 is extracted based on the parallax distribution in the first parallax information I1, and the shape of the three-dimensional object and the distance to the three-dimensional object are estimated. In addition, a history of the distance to the three-dimensional object in the first image P1 is recorded, and the speed of the three-dimensional object is estimated from the history information. Similarly, the area of ​​the three-dimensional object in the second image P2 is extracted based on the parallax distribution in the second parallax information I2, and the shape of the three-dimensional object and the distance to the three-dimensional object are estimated. In addition, a history of the distance to the three-dimensional object in the second image P2 is recorded, and the speed of the three-dimensional object is estimated from the history information.

[0043] Furthermore, in step S41, the object recognition unit 14a also detects light spots in the image P. Here, Fig. 5B(a) is an example of a light spot group detected from the first image P1 with a normal shutter, and Fig. 5B(b) is an example of a light spot group detected from the second image P2 with a low-exposure shutter.

[0044] Next, in step S4a, when multiple light spot groups are extracted in step S41, the object recognition unit 14a judges whether the light spot groups are lamp pairs of the same vehicle from the distances and positions of the respective light spot groups. If it is judged that the light spot groups are lamp pairs of the same vehicle, it outputs coordinate information as the lamp pair and proceeds to step S4b. On the other hand, if it is judged that the light spot groups are not lamp pairs of the same vehicle, it proceeds to step S42.

[0045] In step S4b, the object recognition unit 14a uses the position information of the lamp pairs between the multiple shutters to link three-dimensional object information that is presumed to be the same object. Here, it is assumed that the first three-dimensional object information of the lamp pair L1 illustrated in FIG. 5B(a) is linked to the second three-dimensional object information of the lamp pair L2 illustrated in FIG. 5B(b).

[0046] In step S4c, the object recognition unit 14a determines whether the amount of flare light of the lamp pair is equal to or greater than a threshold value. If the amount of flare light of the lamp pair is equal to or greater than the threshold value, the process proceeds to step S4d. If the amount of flare light is less than the threshold value, the process proceeds to step S42.

[0047] Here, the flare light amount is determined in step S4c, for example, as follows. First, a brightness threshold for extracting a lamp for each exposure amount is determined in advance, and the area of ​​pixels exceeding the brightness threshold is extracted as the lamp (see Fig. 5B (a) and (b)). Next, the lamp sizes W1 and W2 are obtained from the width of each lamp. Then, the difference between the lamp size W1 of the normal shutter (first image P1) and the lamp size W2 of the low exposure shutter (second image P2) is calculated as the flare light amount, and it is determined whether this flare light amount is equal to or greater than the threshold. The method of calculating the difference may be, for example, a method of defining the area obtained by subtracting the circular area area of ​​the lamp size W2 from the circular area area of ​​the lamp size W1 as the flare light amount, as exemplified in Fig. 5B (c).

[0048] In addition, up to this point, the left image P L Or right image P R In the above explanation, it is assumed that one of the left image P is selected and the process of FIG. 5A is performed. L and the right image P R 5A is performed on both the left and right images, the flare light amount may be calculated for each of the left and right images. This allows desired control to be continued using the image captured by the normal camera even if either the left or right camera is out of order.

[0049] In step S4d, the object recognition unit 14a calculates integrated three-dimensional object information by integrating the first three-dimensional object information of the normal shutter (first image P1) and the second three-dimensional object information of the low-exposure shutter (second image P2) linked in step S4b. Specifically, a weighted average of the first three-dimensional object information and the second three-dimensional object information is taken using an integration coefficient set individually for each piece of information (position, speed, shape, etc.) of the three-dimensional object. For example, when integrating the distance to a three-dimensional object, which is one type of three-dimensional object information, if the integration coefficient set for the distance is 0.3, the distance D1 to the three-dimensional object indicated by the first three-dimensional object information and the distance D2 to the three-dimensional object indicated by the second three-dimensional object information are integrated by the following formula 1.

[0050] Distance D after integration = distance D1 × (1-0.3) + distance D2 × 0.3 (Formula 1) In addition, since the first image P1 with a normal shutter and the second image P2 with a low exposure shutter are captured alternately, there is a slight time difference between capturing the two images. Therefore, it is more desirable to correct each piece of information in consideration of the time difference between the capture timing of the first image P1 and the second image P2 and the relative relationship between the target object (the preceding vehicle) and the vehicle itself, and then integrate the information.

[0051] The integration coefficient does not need to be a fixed value, and may be increased or decreased in proportion to the distance to the three-dimensional object, for example. The integration coefficient may be set to a value according to the flare amount by investigating the correlation between the flare amount and the accuracy of each piece of information of the object in advance. Furthermore, when the flare amount is calculated by both the left and right cameras, the variation in the flare amount may also be reflected in the integration coefficient. When the flare amount is large, the weight of the information with higher accuracy, either the normal exposure or the low exposure shutter, is set to be large, thereby improving the accuracy after integration. The integration coefficient may be set individually for each piece of information (position, speed, shape, etc.) that is further subdivided. For example, for the shape, the height and width are set individually. When the flare amount is large or small, the integration coefficient may be set so that the results of either shutter are not used at all during integration.

[0052] In step S42, if the three-dimensional object information is integrated in step S4d, the object recognition unit 14a determines the type of the three-dimensional object using the integrated three-dimensional object information, and if not, determines the type of the three-dimensional object using the first three-dimensional object information obtained from the normal shutter (first image P1).

[0053] Finally, in step S43, the object recognition unit 14a outputs the first three-dimensional object information estimated in step S41 or the integrated three-dimensional object information integrated in step S4d, and the three-dimensional object type determined in step S42 to the downstream vehicle control unit 14b.

[0054] In the stereo camera device 10 of the present embodiment described above, the first three-dimensional object information based on the first image P1, which is easily affected by flare, is integrated with the second three-dimensional object information based on the second image P2, which is less affected by flare, thereby making it possible to increase the reliability of the three-dimensional object information, and thereby to further increase the reliability of the discrimination of the type of three-dimensional object and the reliability of the driving assistance.

[0055] <Specific configuration of object recognition unit 14a> Next, a more specific configuration of the object recognition unit 14a in FIG. 2 will be described with reference to the functional block diagrams of FIG. 6A to FIG. 6C.

[0056] As shown in FIG. 6A, the storage unit 15 outputs a first image P1 and first parallax information I1 captured with a normal shutter, and a second image P2 and second parallax information I2 captured with a low exposure shutter, to the object recognition unit 14a.

[0057] The normal shutter three-dimensional object processing unit 21 in the object recognition unit 14a receives the first normal shutter image P1 and the first parallax information I1. FIG. 6B shows an example of the configuration of the normal shutter three-dimensional object processing unit 21, in which the parallax reference unit 21a, the distance estimation unit 21b, the speed estimation unit 21c, and the shape estimation unit 21d perform three-dimensional object detection in step S41 using the first image P1 and the first parallax information I1, and output the first three-dimensional object information. The lamp detection unit 21e also performs lamp detection in step S41 for the first image P1, and outputs the first lamp information.

[0058] Similarly, the low-exposure shutter second image P2 and the second parallax information I2 are input to the low-exposure shutter three-dimensional object processing unit 22 in the object recognition unit 14a. FIG. 6C shows an example of the configuration of the low-exposure shutter three-dimensional object processing unit 22, in which the parallax reference unit 22a, the distance estimation unit 22b, the speed estimation unit 22c, and the shape estimation unit 22d shown here perform three-dimensional object detection in step S41 using the second image P2 and the second parallax information I2, and output the second three-dimensional object information. In addition, the lamp detection unit 22e performs lamp detection in step S41 for the second image P2, and outputs the second lamp information.

[0059] The lamp pair detection unit 23 performs lamp pair detection of step S4a for the first image P1 based on the first lamp information from the normal shutter three-dimensional object processing unit 21 (see FIG. 5B(a)). Also, the lamp pair detection unit 24 performs lamp pair detection of step S4a for the second image P2 based on the second lamp information from the low exposure shutter three-dimensional object processing unit 22 (see FIG. 5B(b)).

[0060] The lamp detection result linking unit 25 performs a linking process in step S4b on the three-dimensional object information of each lamp pair detected by the lamp pair detection unit 23 and the lamp pair detection unit 24.

[0061] The flare determination unit 26 calculates the amount of flare light by the method illustrated in FIG. 5B(c).

[0062] Three-dimensional object information integrating unit 27 performs the processes of steps S4c and S4d, and if the amount of flare light calculated by flare determining unit 26 is equal to or greater than a predetermined threshold, integrates the first three-dimensional object information from normal shutter three-dimensional object processing unit 21 and the second three-dimensional object information from low-exposure shutter three-dimensional object processing unit 22 under a predetermined rule. On the other hand, if the amount of flare light calculated by flare determining unit 26 is less than the predetermined threshold, the first three-dimensional object information from normal shutter three-dimensional object processing unit 21 is output as is.

[0063] The type determination unit 28 performs the process of step S42 and determines the type of the three-dimensional object based on the target object information output by the three-dimensional object information integrating unit 27 and the recognition dictionary 15c.

[0064] The detection result output unit 29 carries out step S43, and outputs the three-dimensional object information (position, speed, shape, etc.) which is the output of the three-dimensional object information integration unit 27, and the three-dimensional object type which is the output of the type determination unit 28, to the vehicle control unit 14b.

[0065] According to the stereo camera device 10 of the present embodiment described above, even if flare occurs when capturing an image of a three-dimensional object having a high-brightness light source, it is possible to accurately calculate disparity information around the high-brightness light source, and to accurately estimate the position, speed, shape, type, etc. of the three-dimensional object.

[0066] Although the above describes the case where the image processing device of the present invention is a stereo camera device, the image processing device of the present invention may be a monocular camera device that captures images alternately with a normal shutter and a low-exposure shutter. In this case, for example, by adopting a method of assuming the actual size of the image capture target and calculating the distance from the size on the image, it is possible to obtain three-dimensional object information similar to that of a stereo camera device from only the output of the monocular camera, and therefore the above-mentioned three-dimensional object information integration method of the present invention can be used. [Explanation of symbols]

[0067] 10... stereo camera device, 11... camera, 11L... left camera, 11R... right camera, 12... image input interface, 13... image processing unit, 13a... image correction unit, 13b... parallax calculation unit, 14... calculation processing unit, 14a... object recognition unit, 21... normal shutter three-dimensional object processing unit, 22... low exposure shutter three-dimensional object processing unit, 23, 24... lamp pair detection unit, 25... lamp detection result linking unit, 26... flare determination unit, 27... three-dimensional object information integration unit, 28... type determination unit, 29... detection result output unit, 14b... vehicle control unit, 15... memory unit, 15a... image buffer, 15b... parallax buffer, 15c... recognition dictionary, 16... CAN interface, 17... abnormality monitoring unit, CAN... in-vehicle network

Claims

1. a camera that captures a first image with a first exposure amount and captures a second image with a second exposure amount that is less than the first exposure amount; a three-dimensional object extraction unit that extracts a first area in which a three-dimensional object exists from the first image and extracts a second area in which the three-dimensional object exists from the second image; a three-dimensional object information detection unit that detects first three-dimensional object information from the first area and detects second three-dimensional object information from the second area; A three-dimensional object information integration unit that calculates integrated three-dimensional object information by integrating the first three-dimensional object information and the second three-dimensional object information; a flare determination unit that determines a difference between a lamp size of a light source captured in the first image and a lamp size of a light source captured in the second image as a flare light amount captured in the first image; An image processing device comprising:

2. 2. The image processing device according to claim 1, The image processing device, characterized in that the first three-dimensional object information and the second three-dimensional object information include at least one type of information of a distance to the three-dimensional object, a shape of the three-dimensional object, or a speed of the three-dimensional object.

3. 2. The image processing device according to claim 1, The image processing device, characterized in that the camera captures the first image and the second image alternately.

4. 2. The image processing device according to claim 1, The image processing device according to the present invention, wherein the three-dimensional object information integration unit calculates the integrated three-dimensional object information by the following formula: Integrated three-dimensional object information=first three-dimensional object information×(1−integration coefficient)+second three-dimensional object information×integration coefficient

5. 5. The image processing device according to claim 4, 4. An image processing apparatus comprising: an image processing unit configured to process an image of a subject;

6. 6. The image processing device according to claim 5, The three-dimensional object information integration unit When the amount of flare light is equal to or greater than a predetermined threshold, the integrated three-dimensional object information is output. When the amount of flare light is less than the threshold value, the image processing device outputs the first three-dimensional object information.

7. 2. The image processing device according to claim 1, The camera includes: a left camera that captures a first left image of the first exposure amount and a second left image of the second exposure amount; a right camera configured to capture a first right image having the first exposure amount and a second right image having the second exposure amount; having The three-dimensional object extraction unit is a parallax calculation unit that compares the first left image and the first right image to calculate first parallax information and compares the second left image and the second right image to calculate second parallax information; A first three-dimensional object processing unit that extracts the first region in which the three-dimensional object exists from the first image based on the first parallax information; A second three-dimensional object processing unit that extracts the second region in which the three-dimensional object exists from the second image based on the second parallax information; 13. An image processing device comprising:

8. capturing a first image at a first exposure; capturing a second image with a second exposure amount less than the first exposure amount; Extracting a first region in which a three-dimensional object exists from the first image; extracting a second region in which the three-dimensional object exists from the second image; Detecting first three-dimensional object information from the first area; Detecting second three-dimensional object information from the second area; A step of calculating integrated three-dimensional object information by integrating the first three-dimensional object information and the second three-dimensional object information; determining a difference between a lamp size of a light source captured in the first image and a lamp size of a light source captured in the second image as a flare light amount captured in the first image; An image processing method comprising:

Images