External environment recognition device
By acquiring brightness images of different exposure times in multiple shooting devices and generating and synthesizing distance images, the problem of high mismatch rate of synthetic images in the three-dimensional matching of multiple shooting devices is solved, and the accurate determination of stereoscopic objects and dynamic range are realized effective use.
Patent Information
- Application Number
- CN201910084471.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-03-22
- Filing Date
- 2019-01-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-01-29
AI Technical Summary
When multiple shooting devices perform stereo matching, there is a problem of high mismatch rate in the acquisition of synthetic images with high dynamic range, which makes it impossible to adjust the grayscale simultaneously, resulting in mismatch between the parts of the three-dimensional object.
By acquiring brightness images of different exposure times in a plurality of photographing devices, first and second distance images are generated, and blocks with high distance reliability are extracted by the synthetic image generation unit to generate a synthetic image to realize the determination of a three-dimensional object.
In the case of stereo matching, it is possible to effectively determine the stereo object, improve the use efficiency of the dynamic range, reduce the mismatch rate, and ensure the synchronization and accuracy of the image.
Smart Images

Figure CN110298793B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a vehicle exterior environment recognition device for identifying a three-dimensional object existing in the traveling direction of a vehicle. Background Art
[0002] In the past, the following technology is known: detecting three-dimensional objects such as vehicles located in front of the vehicle and controlling the vehicle to avoid collision with the vehicle in front (collision avoidance control) or to maintain a safe distance between the vehicle and the vehicle in front (cruise control) (for example, Patent Document 1).
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent No. 3349060 Summary of the invention
[0006] Technical issues
[0007] In order to realize the collision avoidance control and / or cruise control as described above, it is necessary to appropriately acquire the vehicle exterior environment in front of the vehicle by adjusting the exposure and determine whether the three-dimensional object existing in the traveling direction of the vehicle is a specific object such as the vehicle in front. However, no matter how the exposure is adjusted, the image acquired by the camera may be overexposed in the sun or underexposed in the shade, resulting in a situation where a part of the image cannot be appropriately determined.
[0008] In order to effectively obtain the grayscale of either such a bright image or a dark image, HDR (High Dynamic Range) synthesis that can express a wider dynamic range can be used. However, when the camera is a single camera (monocular), HDR synthesis works effectively, but in stereo matching that requires multiple cameras (compound eyes), if multiple cameras perform HDR synthesis independently, the mismatch rate may increase instead.
[0009] For example, when the dynamic range is temporarily expanded (eg, 256→1024) and then compressed (eg, 1024→256), if the grayscale is adjusted in each image based on noise characteristics, the left and right images cannot be synchronized and parts of the same three-dimensional object do not match each other.
[0010] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a vehicle exterior environment recognition device capable of identifying a three-dimensional object in a wide dynamic range even when performing stereo matching.
[0011] Technical Solution
[0012] In order to solve the above-mentioned problems, the computer of the vehicle exterior environment recognition device of the present invention functions as a first brightness image acquisition unit, a second brightness image acquisition unit, a first distance image generation unit, a second distance image generation unit and a synthetic image generation unit, wherein the first brightness image acquisition unit acquires a plurality of first brightness images captured by a plurality of shooting devices at different positions with a predetermined first exposure time, the second brightness image acquisition unit acquires a plurality of second brightness images captured by a plurality of shooting devices with a second exposure time shorter than the first exposure time, the first distance image generation unit performs pattern matching on the plurality of first brightness images and generates a first distance image, the second distance image generation unit performs pattern matching on the plurality of second brightness images and generates a second distance image, and the synthetic image generation unit extracts a block with high distance reliability from an arbitrary block of the first distance image and a block of the second distance image corresponding to the arbitrary block, and generates a synthetic image.
[0013] It is also possible to configure such that when there is parallax between blocks, the reliability of the distance is higher than when there is no parallax between blocks.
[0014] The reliability of the distance may be higher when the number of blocks whose distance to the block is less than the distance threshold is greater than a predetermined number than when the number of blocks whose distance to the block is less than the distance threshold is less than the predetermined number.
[0015] The configuration may be such that when the accumulated luminance difference of the blocks is equal to or greater than a predetermined value, the reliability of the distance is higher than when the accumulated luminance difference of the blocks is smaller than the predetermined value.
[0016] The configuration may be such that the reliability of the distance is higher for a block having a higher accumulated brightness difference.
[0017] The synthetic image generating unit may be configured to extract the block of the second range image when the accumulated luminance differences of the blocks are the same value.
[0018] Effects of the Invention
[0019] According to the present invention, even when stereo matching is performed, a three-dimensional object can be specified with a wide dynamic range. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is a block diagram showing the connection relationship of the vehicle exterior environment recognition system.
[0021] Figure 2 This is an explanatory diagram showing the relationship between exposure time and image.
[0022] Figure 3 This is a functional block diagram schematically showing the functions of the vehicle exterior environment recognition device.
[0023] Figure 4 This is a flowchart showing the flow of the vehicle exterior environment recognition process.
[0024] Figure 5 It is an explanatory diagram showing a first luminance image.
[0025] Figure 6 It is an explanatory diagram showing the second brightness image.
[0026] Figure 7 It is an explanatory diagram showing a form of generating the first range image.
[0027] Figure 8 It is an explanatory diagram showing a form of generating the second range image.
[0028] Fig. 9 It is an explanatory diagram showing a form of generation of a synthetic image.
[0029] Fig.10 : is a flowchart showing the flow of distance presence / absence determination processing.
[0030] Fig.11 This is an explanatory diagram showing conditions for extracting blocks with high reliability.
[0031] Fig.12 : is a flowchart showing the flow of the brightness difference accumulation determination process.
[0032] Explanation of symbols
[0033] 1: own vehicle 110: camera 120: vehicle exterior environment recognition device 160: first brightness image acquisition unit 162: second brightness image acquisition unit 164: first distance image generation unit 166: second distance image generation unit 168: composite image generation unit 170: three-dimensional object identification unit 214: first distance image 224: second distance image 234: composite image DETAILED DESCRIPTION
[0034] Hereinafter, the preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials and other specific values shown in the embodiment are only for illustration to make the invention easier to understand, and do not limit the present invention unless otherwise specified. It should be noted that in this specification and the accompanying drawings, for elements having substantially the same function and structure, repeated description is omitted by marking the same symbols, and for elements that are not directly related to the present invention, their illustrations are omitted.
[0035] (External environment recognition system 100)
[0036] Figure 11 is a block diagram showing the connection relationship of the vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 includes a camera 110, a vehicle exterior environment recognition device 120, and a vehicle control device (ECU: Engine Control Unit) 130.
[0037] The camera 110 is configured to include camera elements such as CCD (Charge-Coupled Device) and / or CMOS (Complementary Metal-Oxide Semiconductor), and is capable of capturing the exterior environment in front of the vehicle 1 and generating a brightness image (color image or monochrome image) containing at least brightness information. In addition, the camera 110 is arranged at intervals in a substantially horizontal direction on the side of the traveling direction of the vehicle 1 in such a manner that the optical axes of the two camera devices 110 are substantially parallel. The camera 110 continuously generates brightness images obtained by capturing three-dimensional objects in the detection area in front of the vehicle 1 at, for example, 1 / 60 second (60 fps) per frame. Here, the three-dimensional objects recognized by the camera 110 include not only independent objects such as bicycles, pedestrians, vehicles, traffic lights, road signs, guardrails, buildings, etc., but also objects that can be determined as a part of them, such as bicycle wheels.
[0038] In addition, the vehicle exterior environment recognition device 120 obtains brightness images from the two camera devices 110, respectively, and generates a distance image with parallax information using so-called pattern matching, wherein the parallax information includes parallax and image positions indicating the positions of arbitrary blocks in the image. Such pattern matching and distance images will be described in detail later.
[0039] In addition, the vehicle exterior environment recognition device 120 uses the brightness value (color value) based on the brightness image and the three-dimensional space position information in the actual space including the relative distance to the vehicle 1 calculated based on the distance image to first determine the road surface, and group the blocks located on the determined road surface with equal color values and similar three-dimensional position information as three-dimensional objects, thereby determining which specific object (for example, the vehicle in front or a bicycle) the three-dimensional object in the detection area in front of the vehicle 1 corresponds to.
[0040] In addition, after the vehicle exterior environment recognition device 120 identifies the specific object in this way, it controls the vehicle 1 (cruise control) to avoid collision with the specific object (collision avoidance control) or to store the distance to the vehicle in front as a safe distance. It should be noted that the above relative distance is obtained by converting the parallax information of each block in the distance image into three-dimensional position information using the so-called stereo method. Here, the stereo method is a method of deriving the relative distance of the three-dimensional object relative to the camera 110 based on the parallax of the three-dimensional object by using triangulation. Such a method of obtaining three-dimensional position information based on two-dimensional parallax information is also called stereo matching.
[0041] The vehicle control device 130 receives the driver's operation input through the steering wheel 132, the accelerator pedal 134, and the brake pedal 136 and transmits it to the steering mechanism 142, the drive mechanism 144, and the brake mechanism 146, thereby controlling the vehicle 1. In addition, the vehicle control device 130 controls the steering mechanism 142, the drive mechanism 144, and the brake mechanism 146 according to the instruction of the vehicle external environment recognition device 120.
[0042] As described above, in the vehicle exterior environment recognition system 100, the three-dimensional position of a three-dimensional object such as a front vehicle is determined by performing stereo matching based on brightness images from two camera devices 110. However, depending on the image captured by the camera device 110, there may be overexposure in the sunny area and underexposure in the shade, resulting in a situation where a part of the image cannot be obtained and the three-dimensional object located in that place cannot be properly determined.
[0043] Figure 2 is an explanatory diagram showing the relationship between exposure time and image. For example, when entering a tunnel while driving, by setting the exposure time to a long value, the image can be Figure 2 As in (a), the road and / or three-dimensional object in the area 200 in the tunnel can be appropriately obtained. However, in the area 202 in front of the tunnel exit, the sunlight is strong and overexposed. Figure 2 (a) shows the information of area 202.
[0044] Here, if you want to get Figure 2 By setting the exposure time to be short based on the information of area 202 of (a), it is possible to Figure 2 As in (b), the road and / or three-dimensional object in the area 202 ahead of the tunnel exit can be appropriately acquired. Figure 2 The acquired area 200 in the tunnel in (a) is insufficiently exposed and no information can be obtained.
[0045] In order to effectively obtain the grayscale of both the bright image and the dark image, HDR synthesis that can express a wider dynamic range can be used. However, in stereo matching requiring multiple camera devices 110, if HDR synthesis is performed independently by multiple camera devices 110, there is a case where the mismatch rate becomes higher. Therefore, in this embodiment, the shooting method is improved so that a three-dimensional object can be determined with a wide dynamic range even when stereo matching is performed.
[0046] (External environment recognition device 120)
[0047] Figure 3 1 is a functional block diagram showing the general functions of the vehicle exterior environment recognition device 120. Figure 3 As shown, the vehicle exterior environment recognition device 120 is configured to include an I / F unit 150 , a data storage unit 152 , and a central control unit 154 .
[0048] The I / F unit 150 is an interface for bidirectionally exchanging information between the imaging device 110 and the vehicle control device 130. The data storage unit 152 is composed of a RAM, a flash memory, a HDD, etc., and stores various information required for processing by each functional unit described below.
[0049] The central control unit 154 is composed of a semiconductor integrated circuit including a central processing unit (CPU), a ROM storing programs, a RAM as a work area, etc., and controls the I / F unit 150, the data storage unit 152, etc. through the system bus 156. In addition, in the present embodiment, the central control unit 154 also functions as a first brightness image acquisition unit 160, a second brightness image acquisition unit 162, a first distance image generation unit 164, a second distance image generation unit 166, a synthetic image generation unit 168, and a three-dimensional object determination unit 170. Hereinafter, the generation process of the characteristic distance image (synthetic image) of the present embodiment will be described in detail according to the operation of each functional unit of the central control unit 154.
[0050] (External environment recognition and processing)
[0051] Figure 4 1 is a flowchart showing the process of the vehicle exterior environment recognition process. In the vehicle exterior environment recognition process, the first brightness image acquisition unit 160 is used to perform a first brightness image acquisition process (S200) of acquiring two first brightness images captured by the two imaging devices 110 at a predetermined first exposure time, and the second brightness image acquisition unit 162 is used to perform a second brightness image acquisition process (S202) of acquiring two second brightness images captured by the two imaging devices 110 at a second exposure time shorter than the first exposure time.
[0052] Then, the first distance image generating unit 164 performs a first distance image generating process (S204) of pattern matching the plurality of first brightness images to generate a first distance image, and the second distance image generating unit 166 performs a second distance image generating process (S206) of pattern matching the plurality of second brightness images to generate a second distance image.
[0053] Next, the synthetic image generation unit 168 performs a synthetic image generation process (S208) of extracting a block with high distance reliability from two blocks, namely, an arbitrary block of the first range image and a block of the second range image corresponding to the arbitrary block, and generates a synthetic image. Then, the three-dimensional object determination unit 170 performs a three-dimensional object determination process (S210) of determining a three-dimensional object based on the synthetic image. Each process is described in detail below.
[0054] (First Brightness Image Acquisition Process S200)
[0055] The first brightness image acquisition unit 160 sets the exposure time (the time that the shooting element is exposed to light through the lens) of each of the two shooting devices 110 to a predetermined first exposure time (long exposure time) in which a darker part of the external environment of the vehicle (for example, a shaded area) will not be underexposed and its edge can be captured, and acquires the two captured first brightness images.
[0056] Figure 5 It is an explanatory diagram showing a first luminance image. Figure 5 (a) is a first brightness image 210 acquired by the camera 110 disposed on the left side among the camera 110 disposed apart in a substantially horizontal direction. Figure 5 (b) is a first brightness image 212 acquired by the camera 110 disposed on the right side among the camera 110 disposed in a substantially horizontally separated manner. Here, since the image is captured with a relatively long exposure time, although the area 202 in front of the tunnel exit is overexposed due to strong sunlight, the image of the area 200 in the tunnel is appropriately acquired.
[0057] (Second Brightness Image Acquisition Process S202)
[0058] The second brightness image acquisition unit 162 sets the exposure time of each of the two camera devices 110 to a predetermined second exposure time (short exposure time) that does not overexpose the brighter part (e.g., sunny place) in the vehicle exterior environment and can acquire its edge, and acquires the captured multiple second brightness images. In order to suppress exposure, the second exposure time is shorter than the first exposure time. In addition, in order to ensure the simultaneity between the images, the shooting interval between the first brightness image acquisition unit 160 and the second brightness image acquisition unit 162 is preferably a very short time.
[0059] Figure 6 It is an explanatory diagram showing the second brightness image. Figure 6 (a) is a second brightness image 220 acquired by the camera 110 disposed on the left side among the camera 110 disposed apart in the substantially horizontal direction. Figure 6 (b) is a second brightness image 222 acquired by the camera 110 disposed on the right side among the camera 110 disposed in a substantially horizontally separated manner. Here, since the image is captured with a relatively short exposure time, the area 200 in the tunnel is underexposed, but the image of the area 202 in front of the tunnel exit is appropriately acquired.
[0060] (First distance image generation process S204)
[0061] The first distance image generation unit 164 generates one first distance image having parallax information by pattern matching using the two first luminance images 210 and 212 acquired by the first luminance image acquisition unit 160 .
[0062] Figure 7 is an explanatory diagram showing a form of generating the first distance image. The first distance image generating unit 164, for example, Figure 7 The first brightness image 210 shown in (a) is Figure 7 The first brightness image 212 shown in (b) is subjected to pattern matching.
[0063] Specifically, in the two first brightness images 210 and 212, a block corresponding to a block arbitrarily extracted from the other brightness image (for example, an arrangement of 4 pixels horizontally × 4 pixels vertically) is retrieved (pattern matching) from one brightness image, and disparity information including disparity and image position indicating the position of the arbitrary block in the image is derived. Here, horizontal means the image lateral direction of the captured image, and vertical means the image longitudinal direction of the captured image. As the pattern matching, it is considered to compare the brightness (Y) in arbitrary block units between a pair of images. For example, there are methods such as SAD (Sum of Absolute Difference) that obtains the difference in brightness, SSD (Sum of Squared Intensity Difference) that uses the square of the difference, and NCC (Normalized Cross Correlation) that calculates the similarity of the dispersion value obtained by subtracting the average value from the brightness of each pixel.
[0064] The vehicle exterior environment recognition device 120 performs such a block-by-block parallax derivation process on all blocks displayed in the detection area (for example, 600 pixels × 200 pixels). Here, the block is set to 4 pixels × 4 pixels, but the number of pixels in the block can be set arbitrarily. Figure 7 The first distance image 214 as shown in (c).
[0065] Here, in Figure 7 The first brightness image 210 shown in (a) Figure 7 In the first brightness image 212 shown in (b), since the image of the area 200 in the tunnel is properly acquired, Figure 7 In the first distance image 214 of (c), the parallax information of the area 200 in the tunnel is also appropriately derived. However, since the image of the area 202 in front of the tunnel exit is overexposed in the first brightness images 210 and 212, the reliability of the distance in the pattern matching of the area 202 is low.
[0066] (Second distance image generation process S206)
[0067] The second distance image generation unit 166 generates one second distance image having parallax information by pattern matching using the two second luminance images acquired by the second luminance image acquisition unit 162 .
[0068] Figure 8 is an explanatory diagram showing a form of generating the second range image. The second range image generating unit 166 performs, for example, Figure 8 The second brightness image 220 shown in (a) is Figure 8 The pattern matching of the second brightness image 222 shown in (b) is performed. In this way, the following is generated: Figure 8 A second distance image 224 as shown in (c).
[0069] Here, in Figure 8 The second brightness image 220 shown in (a) Figure 8 In the second brightness image 222 shown in (b), since the image of the area 202 ahead of the tunnel exit is properly acquired, Figure 8 In the second distance image 224 of (c), the parallax information of the area 202 further in front of the tunnel exit is also appropriately derived. It should be noted that since the image of the area 200 in the tunnel is underexposed in the second brightness images 220 and 222, the reliability of the distance in the pattern matching of the area 200 is low.
[0070] (Synthetic Image Generation Process S208)
[0071] The synthetic image generation unit 168 uses the first range image 214 and the second range image 224 to extract a block with high distance reliability from two blocks, namely, an arbitrary block in the first range image 214 and a block in the second range image 224 corresponding to the arbitrary block, and generates one synthetic image.
[0072] Fig. 9 is an explanatory diagram showing the generation form of a synthetic image. Fig.10 is a flowchart showing the flow of distance determination processing. Fig.11 is an explanatory diagram showing the conditions for extracting a block with high reliability. Fig.12 : is a flowchart showing the flow of the brightness difference accumulation determination process.
[0073] The synthetic image generation unit 168 uses, for example, Fig. 9 Then, the synthetic image generation unit 168 extracts an arbitrary block 230 from the first distance image 214 shown in (a). Fig. 9 The second distance image 224 shown in (b) extracts a block 232 corresponding to any block 230 of the first distance image 214 (located at the same position in the image). The reliability of the following three parameters is determined for the two blocks 230 and 232, respectively. If all the parameters are determined to have high reliability, the block is determined to have "distance". In addition, if one of the reliability conditions of the parameter is not satisfied, it is determined to have "no distance".
[0074] Here, first, when the block has parallax, the reliability of the distance determination is higher than when the block has no parallax. Fig.10 As shown, the synthetic image generation unit 168 determines whether the block has parallax (S300) for one of the two extracted blocks 230 and 232. As a result, the synthetic image generation unit 168 determines the next parameter when the extracted block has parallax (YES in S300), and does not determine the next parameter when there is no parallax (NO in S300), but determines that it is "no distance" (S302).
[0075] Next, when the number of blocks whose distance to the block is less than the distance threshold is greater than the predetermined number, the reliability of the distance is determined to be high compared to the case where the number of blocks whose distance to the block is less than the distance threshold is less than the predetermined number. Therefore, the synthetic image generation unit 168 first groups the blocks whose horizontal distance x, height y (from the road surface) and relative distance z to the block are less than the predetermined distance threshold (S304). Then, the synthetic image generation unit 168 determines whether the number of blocks (area) of the grouped block group is greater than the predetermined number (S306). As a result, if the number of blocks is greater than the predetermined number (YES in S306), the synthetic image generation unit 168 determines the next parameter, and if the number of blocks is less than the predetermined number (NO in S306), the synthetic image generation unit 168 does not determine the next parameter, but determines it as "no distance" (S302).
[0076] Next, when the accumulated brightness difference (DCDX) of the block is greater than a predetermined value, the reliability of the distance is determined to be high compared to when the accumulated brightness difference of the block is less than the predetermined value. Therefore, the synthetic image generation unit 168 first calculates the accumulated brightness difference of the block (S308). The accumulated brightness difference is a value obtained by accumulating the brightness difference in the horizontal direction (x direction), and can be obtained as follows. For example, it is assumed that the block is a block of 16 pixels, 4 pixels horizontally and 4 pixels vertically. First, the accumulated brightness difference is set to 0. Then, for all 16 pixels, the difference in brightness with the pixel on either side (for example, the pixel on the right) is calculated, and if the difference is greater than a predetermined brightness threshold, the accumulated brightness difference is increased by 1 point. Therefore, the value of the accumulated brightness difference is the number of pixels in the block, and is expressed by 0 to 16 points. And, the higher the number of points, the greater the brightness difference is, that is, the stronger the edge is.
[0077] Next, the synthetic image generation unit 168 determines whether the accumulated brightness difference is greater than a predetermined value (e.g., 1) (S310). As a result, if the accumulated brightness difference of the extracted block is greater than the predetermined value (YES in S310), the synthetic image generation unit 168 determines that "there is a distance" (S312), and if the accumulated brightness difference is less than the predetermined value (NO in S310), the synthetic image generation unit 168 determines that "there is no distance" (S302).
[0078] Next, the synthetic image generation unit 168 determines whether such a distance determination process has been performed on both of the extracted blocks 230 and 232 (S314). As a result, if the distance determination process has been performed on both blocks (YES in S314), the synthetic image generation unit 168 ends the distance determination process. If the distance determination process has not been performed on one block (NO in S314), the synthetic image generation unit 168 repeats the process from step S300 on the other block.
[0079] In this way, after the distance determination is completed for the two extracted blocks 230 and 232, the synthetic image generation unit 168 performs Fig.11 The matrix extracts a block with high reliability from the two extracted blocks 230 and 232.
[0080] Specifically, if Fig.11 (b) Fig.11 As shown in (c) of FIG. 2 , for two blocks 230 and 232, if one block is determined to be “distance-existing” and the other block is determined to be “distance-free”, the synthetic image generation unit 168 determines that the reliability of the distance of the block with “distance-existing” is higher than the reliability of the distance of the block with “distance-free”, and extracts the block determined to be “distance-existing”. Fig.11 As shown in (d) of FIG. 1 , if any one of the blocks is determined to be “no distance”, the synthetic image generating unit 168 does not extract any one of the blocks and sets the block to have no distance information.
[0081] In addition, if Fig.11 As shown in (a) of FIG. 1 , if any of the blocks are determined to be “distant”, the synthetic image generation unit 168 performs a Fig.12 First, the composite image generation unit 168 refers to the brightness difference accumulation of the two blocks determined as "having a distance" to determine whether the values of the brightness difference accumulation are the same value (S350). If the values of the brightness difference accumulation are not the same value (NO in S350), it is considered that the reliability of the distance of the block with a high brightness difference accumulation is higher than the reliability of the distance of the block with a low brightness difference accumulation, and the block with a large brightness difference accumulation value is extracted (S352).
[0082] In addition, if the accumulated values of the brightness differences are the same value (YES in S350), the blocks of the second range image 224 with a short exposure time are preferentially extracted (S354). Here, the reason why the second range image 224 is forcibly extracted when the accumulated brightness differences are the same value is that the longer the exposure time, the more unnecessary light is diffused and acquired as noise, and the image is easily blurred due to the movement of the vehicle under the long exposure time.
[0083] It should be noted that here, an example is given of extracting a block of the second distance image 224 if the accumulated brightness difference value is the same value, but the present invention is not limited to this case. If two blocks are determined to be "distant", the block of the second distance image 224 may be extracted regardless of the accumulated brightness difference value.
[0084] In this way, the composite image generation unit 168 generates, for example, Fig. 9The block 230 of the first range image 214 of (a) is Fig. 9 The block 232 of the second range image 224 of (b) is compared and one block (for example, block 230 here) is extracted. Fig. 9 (c) is a composite image 234.
[0085] Next, the synthetic image generation unit 168 converts the parallax information of each block in the detection area in the synthetic image 234 into three-dimensional position information including a horizontal distance x, a height y, and a relative distance z using the above-mentioned stereo method.
[0086] Here, the disparity information indicates the disparity of each block in the synthesized image 234, whereas the three-dimensional position information indicates the information of the relative distance of each block in the real space. In addition, when the disparity information is derived not in pixel units but in block units, that is, in a plurality of pixel units, the disparity information can be regarded as the disparity information of all pixels belonging to the block and the calculation of the pixel unit can be performed. For such conversion into three-dimensional position information, it is possible to refer to the existing technology such as Japanese Patent Application Laid-Open No. 2013-109391, and therefore the detailed description thereof is omitted here.
[0087] (3D Object Determination Process S210)
[0088] The three-dimensional object determination unit 170 determines a three-dimensional object based on the synthetic image 234 generated by the synthetic image generation unit 168. Specifically, the three-dimensional object determination unit 170 first groups these blocks as three-dimensional objects based on the parallax information in the synthetic image 234 when the horizontal distance x, the height y, and the relative distance z are below a predetermined distance threshold. Then, it is determined which specific object the grouped three-dimensional objects correspond to. For example, when the three-dimensional object has a size, shape, and relative speed similar to that of a vehicle, and a light source such as a brake light or a high-mounted brake light is determined at a predetermined position behind, the three-dimensional object determination unit 170 determines the three-dimensional object as the vehicle in front.
[0089] As described above, in this embodiment, a combination of a plurality of (here, two) brightness images is captured with different exposure times, pattern matching is performed for each exposure time to generate a distance image, and then, the plurality of distance images are synthesized based on the reliability of the distance to generate a synthesized image. Therefore, even when stereo matching is performed, a three-dimensional object can be determined with a wide dynamic range.
[0090] In addition, a program for causing a computer to function as the vehicle exterior environment recognition device 120 and / or a computer-readable storage medium such as a floppy disk, an optical disk, a ROM, a CD, a DVD, or a BD on which the program is recorded is also provided. Here, the program refers to a data processing means described in an arbitrary language or description method.
[0091] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the present invention is not limited to the embodiments. It should be understood that a person skilled in the art can think of various variations or modifications within the scope of the claims, which is self-evident, and these variations or modifications also belong to the technical scope of the present invention.
[0092] For example, in the above-mentioned embodiment, an example of shooting with two different exposure times, the first exposure time and the second exposure time, is cited, but the present invention is not limited to this case. Two brightness images may be generated for three or more exposure times, and pattern matching may be performed to generate three or more distance images. Then, multiple distance images may be synthesized according to the reliability of the distance. With this structure, the exposure time can be subdivided, and a more accurate distance image (synthesized image) can be achieved.
[0093] It should be noted that each step of the vehicle exterior environment recognition process in this specification is not necessarily processed in time series along the order described in the flowchart, and may include parallel processing or sub-process processing.
[0094] Industrial Applicability
[0095] The present invention can be used in a vehicle exterior environment recognition device for identifying a three-dimensional object that exists in the traveling direction of the vehicle.
Claims
1. A vehicle exterior environment recognition device, It is characterized in that The computer of the vehicle exterior environment recognition device functions as a first brightness image acquisition unit, a second brightness image acquisition unit, a first distance image generation unit, a second distance image generation unit, and a composite image generation unit. The first brightness image acquisition unit acquires a plurality of first brightness images captured by a plurality of imaging devices at different positions with a predetermined first exposure time. The second brightness image acquisition unit acquires a plurality of second brightness images captured by the plurality of imaging devices with a second exposure time shorter than the first exposure time. The first distance image generating unit performs pattern matching on the plurality of first brightness images to generate a first distance image. The second distance image generating unit performs pattern matching on the plurality of second brightness images to generate a second distance image. The synthetic image generating unit extracts a block having a high reliability of distance from an arbitrary block of the first range image and a block of the second range image corresponding to the arbitrary block, and generates a synthetic image. When the number of blocks whose distance to the block is less than the distance threshold is greater than a predetermined number, the reliability of the distance is higher than when the number of blocks whose distance to the block is less than the distance threshold is less than the predetermined number.
2. A vehicle exterior environment recognition device, It is characterized in that The computer of the vehicle exterior environment recognition device functions as a first brightness image acquisition unit, a second brightness image acquisition unit, a first distance image generation unit, a second distance image generation unit, and a composite image generation unit. The first brightness image acquisition unit acquires a plurality of first brightness images captured by a plurality of imaging devices at different positions with a predetermined first exposure time. The second brightness image acquisition unit acquires a plurality of second brightness images captured by the plurality of imaging devices with a second exposure time shorter than the first exposure time. The first distance image generating unit performs pattern matching on the plurality of first brightness images to generate a first distance image. The second distance image generating unit performs pattern matching on the plurality of second brightness images to generate a second distance image. The synthetic image generating unit extracts a block having a high reliability of distance from an arbitrary block of the first range image and a block of the second range image corresponding to the arbitrary block, and generates a synthetic image. When the accumulated luminance difference of the block is equal to or greater than a predetermined value, the reliability of the distance is higher than when the accumulated luminance difference of the block is smaller than the predetermined value.
3. A vehicle exterior environment recognition device, It is characterized in that The computer of the vehicle exterior environment recognition device functions as a first brightness image acquisition unit, a second brightness image acquisition unit, a first distance image generation unit, a second distance image generation unit, and a composite image generation unit. The first brightness image acquisition unit acquires a plurality of first brightness images captured by a plurality of imaging devices at different positions with a predetermined first exposure time. The second brightness image acquisition unit acquires a plurality of second brightness images captured by the plurality of imaging devices with a second exposure time shorter than the first exposure time. The first distance image generating unit performs pattern matching on the plurality of first brightness images to generate a first distance image. The second distance image generating unit performs pattern matching on the plurality of second brightness images to generate a second distance image. The synthetic image generating unit extracts a block having a high reliability of distance from an arbitrary block of the first range image and a block of the second range image corresponding to the arbitrary block, and generates a synthetic image. The reliability of the distance is higher for the block with a higher accumulated brightness difference.
4. The vehicle exterior environment recognition device according to claim 3, It is characterized in that The synthetic image generation unit extracts the block of the second range image when the accumulated luminance differences of the blocks are the same value.
