External environment recognition device

Abstract

The present invention provides an external vehicle environment recognition device, which appropriately determines a pair of light sources. The external vehicle environment recognition device includes an image acquisition unit and a pair of light source determination unit. The image acquisition unit acquires a plurality of brightness images obtained by shooting with a shooting unit whose optical axis is set to be different. The pair of light source determination unit extracts a light source (184) from a first brightness image (180a) among the plurality of brightness images, and determines a pair of light sources (186) that form a light source pair based on the positional relationship between the extracted light sources. When the light sources are not paired in the first brightness image, the pair of light sources determination unit extracts a light source from another second brightness image (180b) among the plurality of brightness images, and sets the pair of the light source extracted from the first brightness image and the light source extracted from the second brightness image as the pair of light sources.

Description

Technical Field

[0001] The present invention relates to a vehicle exterior environment recognition device, which can identify a specific object existing in the traveling direction of the vehicle. Background Art

[0002] Conventionally, there are known technologies for detecting a solid object such as a vehicle in front of the vehicle and avoiding a collision with the vehicle ahead (collision avoidance control) or controlling the distance between the vehicle ahead and the vehicle ahead to maintain a safe distance (cruise control) (for example, Patent Document 1). In order to realize such control, for example, there is disclosed a technology for detecting the lighting of a brake light based on a change in brightness or area of ​​a predetermined detection area and determining the deceleration state of the vehicle ahead (for example, Patent Document 2).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 3349060

[0006] Patent Document 2: Japanese Patent No. 3872179 Summary of the invention

[0007] Technical issues

[0008] The taillights and / or brake lights should be arranged symmetrically on the back of the preceding vehicle, and their heights and relative distances from the vehicle should be equal. Therefore, in order to identify the preceding vehicle, it is desirable to identify the taillights and / or brake lights not only as individual light sources, but also as a pair of light sources (hereinafter referred to as a pair of light sources). In this way, by identifying the light sources as a pair of light sources, the accuracy of identifying the preceding vehicle can be improved.

[0009] However, if one of the light sources of the taillight and / or brake light is missing due to external factors such as rain or snow, or the light source area changes due to water seepage, the light source cannot be determined, resulting in reduced accuracy in determining the leading vehicle and delay in speed control relative to the leading vehicle.

[0010] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a vehicle exterior environment recognition device capable of appropriately identifying a light source.

[0011] Technical Solution

[0012] In order to solve the above-mentioned problem, the vehicle exterior environment recognition device of the present invention comprises: an image acquisition unit, which acquires multiple brightness images obtained by setting the optical axis to different shooting units; and a light source determination unit, which extracts a light source from a first brightness image among the multiple brightness images, and determines a pair of light sources that form a light source pair based on the positional relationship between the extracted light sources. When the light sources are not paired in the first brightness image, the light source determination unit extracts a light source from another second brightness image among the multiple brightness images, and sets the pair of the light source extracted from the first brightness image and the light source extracted from the second brightness image as a pair of light sources.

[0013] A light source tracking unit may be provided for tracking each of the light sources extracted by the pair light source identification unit, and the pair light source identification unit may determine that the light sources are not paired based on the tracking result of the light sources.

[0014] Technical Effects

[0015] According to the present invention, it is possible to appropriately determine the light source. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram showing the connection relationship of the vehicle exterior environment recognition system.

[0017] Figure 2 This is a functional block diagram showing the general functions of the vehicle exterior environment recognition device.

[0018] Figure 3 This is a flowchart showing the flow of the vehicle exterior environment recognition method.

[0019] Figure 4 This is an explanatory diagram for explaining a brightness image.

[0020] Figure 5 This is an explanatory diagram for explaining a distance image.

[0021] Figure 6 This is an explanatory diagram for explaining the light source determination process.

[0022] Figure 7 This is an explanatory diagram for explaining the light source determination process.

[0023] Figure 8 This is an explanatory diagram for explaining three-dimensional object identification processing.

[0024] Explanation of symbols

[0025] 110 Camera

[0026] 120 External environment recognition device

[0027] 160 Image acquisition unit

[0028] 162 Distance Image Generation Unit

[0029] 164 Light source determination unit

[0030] 166 Light Source Tracking Department

[0031] 168 Three-dimensional Object Determination Department

[0032] 180a First brightness image

[0033] 180b Second brightness image

[0034] 182 Distance Image DETAILED DESCRIPTION

[0035] Hereinafter, the preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The dimensions, materials and other specific numerical values ​​shown in the embodiments are only examples for making the invention easy to understand, and unless otherwise specified, they do not limit the present invention. It should be noted that in this specification and the accompanying drawings, the same symbols are marked for elements having substantially the same functions and structures, thereby omitting repeated descriptions, and in addition, the illustrations of elements that are not directly related to the present invention are omitted.

[0036] (External environment recognition system 100)

[0037] Figure 1 1 is a block diagram showing the connection relationship of the vehicle exterior environment recognition system 100. The vehicle exterior environment recognition system 100 includes a camera 110, a vehicle exterior environment recognition device 120, and a vehicle control device (ECU: Electronic Control Unit) 130.

[0038] The camera 110 is configured to include a camera element such as a CCD (Charge-Coupled Device) and a CMOS (Complementary Metal-Oxide Semiconductor), and is capable of capturing the exterior environment in front of the vehicle 1 and generating a brightness image (a color image or a monochrome image) containing at least brightness information. In addition, the camera 110 is configured to be separated in a substantially horizontal direction on the side of the travel direction of the vehicle 1 in such a manner that the optical axes of the two camera devices 110 are substantially parallel. The camera 110 continuously generates a brightness image of a three-dimensional object existing in a detection area in front of the vehicle 1 at, for example, 1 / 60 seconds per frame (60 fps).

[0039] In addition, the vehicle exterior environment recognition device 120 obtains brightness images from the two camera devices 110, and recognizes the environment outside the vehicle, such as the behavior of the preceding vehicle and / or the oncoming vehicle, the condition of the road, etc., based on the two brightness images. Furthermore, the vehicle exterior environment recognition device 120 controls the driving (speed, steering angle) of the vehicle 1 based on the recognized vehicle exterior environment and the driving condition of the vehicle 1. The vehicle exterior environment recognition device 120 will be described in detail later.

[0040] The vehicle control device 130 receives the driver's operation input through the steering wheel 132, the accelerator pedal 134, and the brake pedal 136, and controls the vehicle 1 by transmitting the input to the steering mechanism 142, the drive mechanism 144, and the brake mechanism 146. In addition, the vehicle control device 130 controls the steering mechanism 142, the drive mechanism 144, and the brake mechanism 146 according to the instruction of the vehicle external environment recognition device 120.

[0041] (External environment recognition device 120)

[0042] Figure 2 1 is a functional block diagram showing the general functions of the vehicle exterior environment recognition device 120. Figure 2 As shown, the vehicle exterior environment recognition device 120 is configured to include an I / F unit 150 , a data holding unit 152 , and a central control unit 154 .

[0043] The I / F unit 150 is an interface for bidirectionally exchanging information with the imaging device 110 and the vehicle control device 130. The data storage unit 152 is composed of a RAM, a flash memory, a HDD, etc., and stores various information required for processing of each functional unit described below.

[0044] The central control unit 154 is composed of a semiconductor integrated circuit including a central processing unit (CPU), a ROM storing programs, a RAM as a work area, etc., and controls the I / F unit 150, the data holding unit 152, etc. through the system bus 156. In addition, in the present embodiment, the central control unit 154 also functions as an image acquisition unit 160, a distance image generation unit 162, a light source determination unit 164, a light source tracking unit 166, and a three-dimensional object determination unit 168. The following is also based on the operation of each functional unit of the central control unit 154. The vehicle exterior environment recognition method of identifying the light source in front of the vehicle 1 and determining the area of ​​the three-dimensional object (for example, the preceding vehicle) that is characteristic of the present embodiment is described in detail.

[0045] (External vehicle environment recognition method)

[0046] Figure 3It is a flowchart representing the process of the vehicle exterior environment recognition method. The flowchart is executed at predetermined interruption times. In the vehicle exterior environment recognition method, first, the image acquisition unit 160 acquires a plurality of brightness images (S200). Next, the distance image generation unit 162 generates a distance image (S202). Next, the light source determination unit 164 extracts light sources from the brightness image, and determines a pair of light sources as a light source pair based on the positional relationship between the extracted light sources (S204 to S214). Next, the light source tracking unit 166 tracks the extracted light sources respectively (S216). Next, the three-dimensional object determination unit 168 determines a three-dimensional object based on the pair of light sources (S218). Below, each process of the vehicle exterior environment recognition method is described in detail, and the description of processes that are not related to the features of this embodiment is omitted.

[0047] (Image acquisition processing S200)

[0048] Figure 4 1 is an explanatory diagram for explaining a brightness image. The image acquisition unit 160 acquires a plurality of (here, two) brightness images captured by the imaging device 110 with different optical axes. Here, it is assumed that the image processing unit 160 acquires Figure 4 The first brightness image 180a and the first brightness image 180b captured by the camera 110 located on the relatively right side of the vehicle 1 are shown in (a) and Figure 4 The second brightness image 180 b shown in (b) of FIG. 1 , which is captured by the camera 110 located on the relatively left side of the vehicle 1 , is used as the brightness image 180 .

[0049] It should be noted that, here, the brightness image 180 captured by the camera 110 located on the relatively right side of the vehicle 1 is exemplified as the first brightness image 180a, and the brightness image 180 captured by the camera 110 located on the relatively left side of the vehicle 1 is exemplified as the second brightness image 180b. However, the brightness image 180 captured by the camera 110 located on the relatively left side of the vehicle 1 may be used as the first brightness image 180a, and the brightness image 180 captured by the camera 110 located on the relatively right side of the vehicle 1 may be used as the second brightness image 180b.

[0050] If reference Figure 4 , it can be understood that: due to the different shooting positions of the shooting device 110, the image positions of the three-dimensional objects contained in the first brightness image 180a and the second brightness image 180b are different in the horizontal direction. Here, horizontal means that the screen of the captured image is horizontal, and vertical means that the screen of the captured image is vertical.

[0051] It should be noted that the three-dimensional objects included in the brightness image include not only independent objects such as vehicles located in front of the vehicle 1 (front vehicles, oncoming vehicles), bicycles, pedestrians, traffic lights, roads (travel roads), road signs, guardrails, buildings, etc., but also objects that can be identified as part of the three-dimensional objects, such as the vehicle's rear lights and tail lights.

[0052] (Distance image generation process S202)

[0053] Figure 5 is an explanatory diagram for explaining a distance image. The distance image generation unit 162 generates a distance image based on the image acquired by the image acquisition unit 160. Figure 5 The first brightness image 180a shown in (a) and Figure 5 The second brightness image 180b shown in (b) is used to generate Figure 5 A distance image 182 as shown in (c).

[0054] Specifically, the distance image generating unit 162 uses so-called pattern matching to derive disparity information including disparity and image position indicating the position of an arbitrary block within the image, wherein the pattern matching retrieves a block corresponding to a block arbitrarily extracted from another brightness image (here, the first brightness image 180a) (for example, an arrangement of 4 pixels horizontally × 4 pixels vertically) from one brightness image (here, the second brightness image 180b).

[0055] As a pattern matching method, it is considered to compare the brightness (Y) between a pair of images in units of arbitrary blocks. For example, there are methods such as SAD (Sum of Absolute Difference) which takes the difference in brightness, SSD (Sum of Squared intensity Difference) which uses the square of the difference, and NCC (Normalized Cross Correlation) which takes the similarity of the variance obtained by subtracting the average value from the brightness of each pixel.

[0056] The distance image generation unit 162 performs such a parallax derivation process in units of blocks for all blocks displayed in the detection area (for example, 600 pixels×200 pixels). Here, the block is set to 4 pixels×4 pixels, but the number of pixels in the block can be set arbitrarily.

[0057] However, the distance image generation unit 162 can derive the parallax for each block as the detection resolution unit, but cannot identify what kind of object the block is a part of. Therefore, the parallax information is not derived for each object unit, but is independently derived for each detection resolution unit (e.g., block unit) within the detection area. Here, for the sake of convenience, the block from which the parallax is derived is represented by a black dot.

[0058] The distance image generation unit 162 converts the parallax information of each block in the distance image 182 into three-dimensional position information using a so-called stereo method, and derives a relative distance. Here, the stereo method is a method of deriving the relative distance of the object with respect to the camera 110 based on the parallax of the object using a triangulation method.

[0059] (Light Source Determination Processing S204 to S214)

[0060] In the vehicle exterior environment recognition device 120, blocks with equal color values ​​and similar three-dimensional position information are grouped as objects using the brightness value (color value) based on the brightness image 180 and the three-dimensional position information in the real space calculated based on the distance image 182 and including the relative distance to the host vehicle 1. And, it is determined which specific object (for example, the preceding vehicle) the object in the detection area in front of the host vehicle 1 corresponds to. Here, the processing of determining the light source, which is an element for determining such a specific object, especially the preceding vehicle, is described.

[0061] Figure 6 , Figure 7 FIG. 2 is an explanatory diagram for explaining the light source determination process. Figure 6 As shown in (a), the light source determination unit 164 extracts each light source (single light source) 184 having brightness and color distribution equivalent to a tail light or a brake light from one brightness image 180 (for example, the first brightness image 180a) among the plurality of brightness images 180 (S204).

[0062] At this time, if the brightness of the light source 184 is too high, the imaging device 110 shortens the exposure time to generate the brightness image 180. In addition, the imaging device 110 may alternately generate the brightness image 180 with different exposure times, and use the brightness image 180 with a relatively short exposure time for the light source determination unit 164 to extract the light source 184 based on the brightness and color distribution.

[0063] The light source determination unit 164 removes the following light source 184 from the candidates of the tail light or the brake light, that is, the light source 184 is a light source 184 that has no distance or cannot be determined in the distance image 182 even if it can be identified as the light source 184 based on the brightness and color distribution. It should be noted that a reflector as a three-dimensional object may have brightness and color distribution similar to that of a tail light or a brake light. The light source determination unit 164 also removes the following light source 184 from the candidates of the tail light or the brake light, that is, the light source 184 is an object with a predetermined size and / or shape such as a reflector even if it can be identified as the light source 184 based on the brightness and color distribution.

[0064] The light sources 184 that are candidates for tail lights or brake lights and have not been eliminated are registered as tracking targets (light trails) of the light source tracking unit 166 described later.

[0065] Next, the light source determination unit 164 refers to the distance image 182 and determines a pair of light sources (S206) that are light sources having a position (horizontal position, vertical position, relative distance) equivalent to the tail light or brake light based on the positional relationship between the light sources 184 that are candidates for the tail light or brake light and have not been removed.

[0066] Specifically, the light source determination unit 164 accumulates the points of the two light sources 184 according to each of the following four conditions, and sets the combination with the highest possibility of pairing as the light source 186. The four conditions are: (1) the horizontal distance is less than the legal length of the vehicle, (2) the pixels overlap each other in the vertical direction, (3) the difference in relative distance is within a predetermined range, and (4) the difference in area is within a predetermined range. Figure 6 In the example (b), a pair of two light sources 184 is determined as a pair of light sources 186 .

[0067] It should be noted that the light source determination unit 164 reads the light source 186 set as the light source in the previous frame (or the previous frame), and when the combination of the light source 186 set as the light source in this frame is different from the light source 186 in the previous frame, the light source determination unit 164 compares the points obtained based on the above four conditions, and determines the one with a higher point number (most suitable as the light source) as the light source 186. At this time, when the light source 186 in the current frame has a higher point number, the light source determination unit 164 clears the information of the light source 186 in the previous frame, and registers the light source 186 in the current frame as the tracking target of the light source tracking unit 166.

[0068] In this way, by identifying the light source 184 that is a candidate for a tail light or a brake light as the counter light source 186 , it is possible to improve the accuracy of identifying the preceding vehicle.

[0069] However, if, for example, one of the light sources 184 of the taillight and / or brakelight is missing due to external factors such as rain or snow, or the area of ​​the light source 184 is different from the original area due to water seepage, the light source determination unit 164 will not be able to determine the light source 186 that should have existed. This will result in a decrease in the accuracy of determining the leading vehicle and a delay in the speed control relative to the leading vehicle. Therefore, in this embodiment, the second brightness image 180b different from the first brightness image 180a in which the light source 184 is missing is used to extract the light source 184, and the light source 186 is appropriately determined.

[0070] Specifically, the light source determination unit 164 determines whether there is a paired light source 184 in the first brightness image 180a (S208). And if there is no paired light source 184 ("No" in S208), the light source determination unit 164 extracts the light source 184 from another second brightness image 180b in the plurality of brightness images 180 (S210), and sets the light source 184 extracted from the first brightness image 180a and the light source 184 extracted from the second brightness image 180b as a paired light source 186 (S212). In addition, if there is a paired light source 184 ("Yes" in S208), the light source determination unit 164 does not perform steps S210 and S212, but proceeds to the next process.

[0071] Here, it is assumed that the image acquisition unit 160 acquires Figure 7 (a) of the first brightness image 180a and Figure 7 (b) is a second brightness image 180b.

[0072] The light source determination unit 164 extracts the light source 184 from the first brightness image 180a. Figure 7 As shown by the arrow symbol in (a), the light source determination unit 164 refers to the tracking result of the light source 184 being tracked by the light source tracking unit 166 described later, and cannot extract the light source 184 that has been determined as the target light source 186 in the previous frame, and determines that the target light source 186 does not exist. The light source determination unit 164 refers to the second brightness image 180b acquired at the same time as the first brightness image 180a, and as shown in FIG. Figure 7 As shown by the arrow in (b), the light source 184 is extracted from the second brightness image 180b.

[0073] However, although the distance image 182 having three-dimensional position information (horizontal position, vertical position, relative distance) is Figure 5The positional relationship shown is generated corresponding to the first brightness image 180a, but the positional relationship of the distance image 182 does not correspond to the second brightness image 180b. Therefore, the light source determination unit 164 uses the so-called stereo method to convert the three-dimensional position information of the light source 184 determined as the light source 186 in the previous frame into the three-dimensional position information in the second brightness image 180b.

[0074] The light source determination unit 164 extracts the light source 184 in the second brightness image 180b based on the brightness and color distribution of the three-dimensional object existing around the converted three-dimensional position information. Furthermore, the light source determination unit 164 converts the three-dimensional position information of the light source 184 extracted in the second brightness image 180b into the three-dimensional position information in the first brightness image 180a using the so-called stereo method. Figure 7 As shown by the arrow symbol in (c), the light source 184 is corresponding to the first brightness image 180a.

[0075] The paired light source determination unit 164 sets the light source 184 extracted from the first luminance image 180 a and the light source 184 extracted from the second luminance image 180 b and corresponding to the first luminance image 180 a as paired light sources 186 .

[0076] It should be noted that if the number of times such a pair of light sources 186 is continuously extracted as the same object is a predetermined number of times (for example, 5 times) or more, the reliability of the pair of light sources 186 and / or the determination value of the vehicle similarity of the preceding vehicle including the pair of light sources 186 can be improved. For example, when one light source 184 becomes a pair candidate of multiple light sources 184, it will be paired with a light source with a high reliability as the pair of light sources 186.

[0077] Before the light source determination unit 164 determines whether all the light sources 184 extracted from the first brightness image 180a as candidates for tail lights or brake lights are the opposing light sources 186 ("No" in S214), it repeatedly executes from step S206. If it is determined whether all the light sources 184 are the opposing light sources 186 ("Yes" in S214), it proceeds to the next light source tracking processing S216.

[0078] (Light source tracking process S216)

[0079] The light source tracking unit 166 tracks the light source 184 extracted as a candidate of a tail light or a brake light by the light source determination unit 164 based on the position information, speed, and rotation speed of the light source 184. At this time, if the target light source 184 has been tracked (detected) in the previous frame, it is associated with the light source 184 of the previous frame based on the ratio of overlap of pixels of the light source 184 and the similarity of distance (based on the level of the accumulated points). In addition, the light source tracking unit 166 preferentially tracks the light source 184 with a high accumulated point count over the light source 184 with a low accumulated point count.

[0080] (3D Object Determination Process S218)

[0081] Figure 8 : is an explanatory diagram for explaining the three-dimensional object determination processing. First, the three-dimensional object determination unit 168 groups the blocks that are at equal relative distances to the host vehicle 1 and close to each other in the vertical and horizontal directions among the multiple blocks located at a height above a predetermined distance from the road surface in the distance image 182, and sets them as candidates for the leading vehicle, for example. Next, the three-dimensional object determination unit 168 determines a rectangular area in the first brightness image 180a that includes the entire three-dimensional object determined in this way as a three-dimensional object area 190. Here, the rectangle is composed of two straight lines extending in the vertical direction and tangent to the left and right edges of the three-dimensional object, respectively, and two straight lines extending in the horizontal direction and tangent to the upper and lower edges of the three-dimensional object, respectively.

[0082] The three-dimensional object determination unit 168 determines the three-dimensional object in the three-dimensional object region 190 as a specific object, that is, a preceding vehicle, based on the positional relationship between the three-dimensional object region 190 and the opposing light source 186 formed in this way. Specifically, when the horizontal width and height of the three-dimensional object region 190 are equivalent to those of the vehicle, and the difference in relative distance between the three-dimensional object region 190 and the opposing light source 186 is within a predetermined range and the opposing light source 186 is located at an appropriate height in the three-dimensional object region 190, the three-dimensional object determination unit 168 determines the three-dimensional object in the three-dimensional object region 190 as a preceding vehicle.

[0083] In this way, if the three-dimensional object area 190 is determined to be the preceding vehicle, the vehicle exterior environment recognition device 120 can control the host vehicle 1 to avoid collision with the preceding vehicle (collision avoidance control) or maintain the inter-vehicle distance with the preceding vehicle at a safe distance (cruise control).

[0084] In this embodiment, since the light source 186 can be appropriately determined based on the second brightness image 180 b , the preceding vehicle can be stably determined, and a decrease in the accuracy of determining the preceding vehicle and a delay in speed control relative to the preceding vehicle can be avoided.

[0085] In addition, a program for causing a computer to function as the vehicle exterior environment recognition device 120 and a computer-readable storage medium such as a floppy disk, magneto-optical disk, ROM, CD, DVD, BD, etc. storing the program are provided. Here, the program refers to data processing means written in any language or writing method.

[0086] The preferred embodiment of the present invention has been described above with reference to the accompanying drawings, but the present invention is certainly not limited to this embodiment. It should be understood that it is obvious that a person skilled in the art can think of various variations or modifications within the scope of the claims, and these variations or modifications certainly also belong to the technical scope of the present invention.

[0087] It should be noted that the steps of the vehicle exterior environment recognition method of the present specification do not necessarily have to be processed in a time series in the order described in the flowchart, and may also include processing in parallel or by a subroutine.

[0088] Industrial availability

[0089] The present invention can be used for a vehicle exterior environment recognition device that identifies a specific object that exists in the traveling direction of the vehicle.

Claims

1. A vehicle exterior environment recognition device, characterized in that: have: an image acquisition unit that acquires a plurality of brightness images captured by an imaging unit with different optical axes; and a pair of light sources determining unit, which extracts light sources from a first brightness image among the plurality of brightness images, and determines a pair of light sources forming a light source pair based on a positional relationship between the extracted light sources. When the light sources are not paired in the first brightness image, the paired light source determination unit extracts a light source from another second brightness image among the plurality of brightness images, and sets a pair of the light source extracted from the first brightness image and the light source extracted from the second brightness image as a paired light source.

2. The vehicle exterior environment recognition device according to claim 1, characterized in that: have: a light source tracking unit, which tracks the light sources extracted by the light source determination unit respectively; The paired light source specifying unit determines that the light sources are not paired based on a tracking result of the light sources.

Images