Signal recognition device

Abstract

To appropriately recognize a traffic light for a self-vehicle lane even in a weak-light environment.SOLUTION: A traffic light recognition device comprises: an imaging apparatus 11 which images a front area of a vehicle, thereby acquiring image data; a light irradiation device 12 which irradiates a predetermined range including the front area of the vehicle with light and receiving reflected waves from a three-dimensional object being present within the range, thereby acquiring three-dimensional object information including a relative position and a shape of the three-dimensional object; and a control unit 10 which extracts an area where there is a possibility that a lamp is present, as a lamp area based on the image data, determines whether or not a traffic light is present in the lamp area based on the three-dimensional object information and, in a case where it is determined that the traffic light is present in the lamp area, based on the three-dimensional object information, determines whether or not the traffic light is a traffic light for a self-vehicle lane.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a traffic light recognition device capable of recognizing a traffic light. [Background technology]

[0002] There have been known traffic light recognition devices capable of recognizing traffic lights (including the color of the lights, etc.). For example, the traffic light recognition device of Patent Document 1 (hereinafter also referred to as the "conventional device") is configured to capture an image of a traffic light with an imaging device, detect chromaticity information and luminance information of the lights of the traffic light (hereinafter also referred to as the "signal light") from the captured traffic light image, and determine the color of the signal light based on the chromaticity information and luminance information, thereby recognizing the color of the signal light. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-015759 A Summary of the Invention

[0004] When recognizing a traffic light based on an image captured by an imaging device as in the conventional device, it is difficult to capture the casing of the traffic light in a weak light environment (relatively dark environment) such as at night. Therefore, in a weak light environment, the presence or absence of a traffic light must be determined based only on the color of the light included in the captured image, but with this configuration, there is a possibility that lights other than signal lights (for example, street lights and tail lights of a preceding vehicle) may be erroneously recognized (detected) as traffic lights. In addition, since the captured image is a two-dimensional image, the estimation accuracy of the positional relationship between the vehicle and the traffic light is not very high. Therefore, even if a traffic light can be recognized based on the captured image, there is a possibility that a traffic light for a lane intersecting the own lane (including lanes at a grade separation) may be erroneously selected as the traffic light for the own lane, rather than the traffic light for the own lane. In other words, a configuration for recognizing a traffic light based on a captured image has a problem of erroneous detection or selection of a traffic light, especially in a weak light environment.

[0005] The present invention has been made to address the above-mentioned problems. That is, one of the objects of the present invention is to provide a traffic light recognition device that can properly recognize a traffic light for the own lane even in a weak light environment.

[0006] The traffic light recognition device according to the present invention (hereinafter referred to as the "device of the present invention") comprises: Mounted on the vehicle, an imaging device (11) capable of acquiring image data by imaging an area in front of the vehicle; a light projection device (12) that projects light onto a predetermined range including the forward area of ​​the vehicle and receives reflected waves from a three-dimensional object present within the predetermined range, thereby acquiring three-dimensional object information including a relative position and a shape of the three-dimensional object; Based on the image data, an area where a light may exist is extracted as a light area (step 210); Based on the solid object information, it is determined whether or not a traffic light is present in the light area (step 215); When it is determined that a traffic light is present in the light area, it is determined whether or not the traffic light is a traffic light for the own lane based on the three-dimensional object information (step 225). A control unit (10) configured as described above; Equipped with. The control unit (10) generating a captured image as a color image based on the image data; Calculate the RGB values ​​of each pixel that constitutes the color image; For each pixel, determining whether the calculated RGB value is within a predetermined range; If the distance between pixels whose RGB values ​​are determined to be within the range is equal to or less than a predetermined distance threshold, a rectangular area including the periphery of these pixels and below them is extracted as the light area.

[0007] The device of the present invention determines the presence or absence of a traffic light based on three-dimensional object information acquired by a light irradiation device. The three-dimensional object information can be acquired regardless of the brightness of the surroundings. In addition, the three-dimensional object information is three-dimensional information including the relative position and shape of the three-dimensional object. Therefore, compared to a configuration that recognizes traffic lights based on a two-dimensional captured image, the device can accurately recognize traffic lights even in a weak light environment, and the possibility of erroneously detecting or selecting a traffic light can be significantly reduced. With this configuration, it is possible to properly recognize the traffic light for the vehicle's own lane even in a weak light environment.

[0008] In particular, the device of the present invention is configured to first extract a light area based on image data and determine whether a traffic light exists in the light area, rather than determining whether all three-dimensional objects included in the three-dimensional object information are traffic lights or not. This is based on the knowledge that a traffic light is a type of light and therefore is highly unlikely to exist in an area other than the light area. With this configuration, the number of three-dimensional objects to be determined as to whether a traffic light exists or not can be significantly reduced, thereby reducing the processing load without reducing the accuracy of traffic light recognition. Furthermore, in the configuration of the device of the present invention, not only the surroundings of the pixel group whose RGB values ​​are within a predetermined range are included in the light area, but also the area below it, which increases the likelihood that part of the traffic light pole is included in the light area, thereby further improving the recognition accuracy of the traffic light.

[0009] Furthermore, when controlling the acceleration / deceleration of a vehicle based on a recognized traffic light, the distance to the traffic light can be accurately calculated because the three-dimensional object information is three-dimensional. Therefore, when multiple traffic lights are recognized, the timing at which the vehicle will encounter a traffic light can be accurately calculated based on the distance to the traffic light. Therefore, the control timing and control amount for acceleration / deceleration can be appropriately calculated based on the distance to the traffic light and the color of the signal light.

[0010] One aspect of the present invention is The device further includes a storage device that prestores housing information including the shapes of the housings of various types of traffic lights. The control unit (10) calculating an approximation of a three-dimensional object included in the three-dimensional object information in an area corresponding to the light area to a casing of a traffic light based on the casing information, and determining that a traffic light exists in the light area when the approximation is equal to or greater than a predetermined approximation threshold value; It is structured as follows.

[0011] Since the three-dimensional object information is three-dimensional, the shape of the three-dimensional object can be acquired with high accuracy. Therefore, when the approximation of the three-dimensional object to the "traffic light housing included in the housing information" is equal to or greater than the approximation threshold, it means that there is an extremely high possibility that the three-dimensional object is the housing of the traffic light. Therefore, with this configuration, it is possible to determine with high accuracy whether or not a traffic light exists in the light area.

[0012] In one aspect of the invention, The light irradiation device is a three-dimensional Lidar. do.

[0013] In the above description, in order to facilitate understanding of the invention, the symbols used in the embodiments are added in parentheses to the constituent elements of the invention corresponding to the embodiments, but each constituent element of the invention is not limited to the embodiment defined by the said symbols. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic diagram of a traffic light recognition device (embodiment device) according to an embodiment of the present invention; [Diagram 2] 11 is a diagram showing a processing procedure executed by a CPU of a traffic light recognition ECU of the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] (composition) Hereinafter, a traffic light recognition device (hereinafter also referred to as "the present embodiment device") according to an embodiment of the present invention will be described with reference to the drawings. The present embodiment device is mounted on a vehicle. The present embodiment device is configured to recognize a traffic light for the vehicle's own lane, accelerate and decelerate the vehicle based on the color of the light and the distance to the traffic light, thereby executing automatic driving control or driving assistance control. Hereinafter, a vehicle equipped with the present embodiment device will be referred to as "the vehicle itself." Herein, "the vehicle itself" means the lane in which the vehicle itself is currently traveling. In addition, in a country where traffic lights are installed on a roadway basis including multiple lanes with the same traveling direction, the "traffic light for the vehicle itself" means a vehicular traffic light installed to facilitate traffic on the roadway including the vehicle itself and lanes with the same traveling direction as the vehicle itself. On the other hand, in a country where traffic lights are installed on a lane basis, the "traffic light for the vehicle itself" means a vehicular traffic light installed to facilitate traffic on the roadway including the vehicle itself.

[0016] As shown in Fig. 1, the present embodiment includes a traffic light recognition ECU 10, and a camera sensor 11, a Lidar sensor 12, a display device 20, a drive device 30, and a braking device 40 connected thereto. The traffic light recognition ECU 10 includes a microcomputer as a main component. ECU is an abbreviation for Electronic Control Unit. The microcomputer includes a CPU, ROM, RAM, an interface (I / F), etc., and the CPU realizes various functions by executing instructions (programs, routines) stored in the ROM.

[0017] The traffic light recognition ECU 10 is configured to acquire signals output from the camera sensor 11 and the Lidar sensor 12 at predetermined time intervals, and control the display device 20, the drive device 30, and the braking device 40 based on the acquired signals. Hereinafter, the traffic light recognition ECU 10 is also simply referred to as "ECU 10."

[0018] The camera sensor 11 (imaging device) is installed on the rear of the room mirror (inner mirror / rearview mirror) of the vehicle. The camera sensor 11 is a visible light camera that captures an image of the area ahead of the vehicle (in other words, captures an image of a three-dimensional object present in the area ahead) and outputs image data obtained by capturing the image to the ECU 10. The three-dimensional object includes a traffic light, a vehicle, a street lamp, a median strip, a building, etc.

[0019] The Lidar sensor 12 (light irradiation device) is a three-dimensional Lidar sensor and is housed in a dome provided on the roof of the vehicle. Lidar is an abbreviation of Light Detection and Ranging. The Lidar sensor 12 irradiates a pulsed near-infrared wavelength band laser light in a 360° area around the vehicle, and receives reflected light from the three-dimensional object if a three-dimensional object exists. The Lidar sensor 12 calculates the presence or absence of a three-dimensional object and the relative position and shape of the three-dimensional object (in other words, a spatial coordinate group of the three-dimensional object) based on the irradiation timing and reception timing of the laser light. The relative position of the three-dimensional object includes the distance from the vehicle to the three-dimensional object and the orientation of the three-dimensional object relative to the vehicle. The shape of the three-dimensional object includes the height of the three-dimensional object and the orientation (posture) of the three-dimensional object. The Lidar sensor 12 outputs the calculation result to the ECU 10 as three-dimensional point cloud information. The three-dimensional point cloud information corresponds to an example of "three-dimensional object information".

[0020] In this embodiment, a solid-state sensor is used as the Lidar sensor 12, but the type of Lidar sensor is not limited to this, and for example, a raster scan type or a wobbling scan type sensor may be used. In addition, the sensor that acquires the three-dimensional point cloud information is not limited to the Lidar sensor 12, and may be any sensor that can accurately calculate the relative position and shape of a three-dimensional object in three dimensions. In addition, the Lidar sensor 12 is not limited to a configuration that irradiates laser light 360° around the vehicle, but may be configured to irradiate laser light to an area that includes at least the forward area.

[0021] The display device 20 is a device for displaying information about the color of a signal lamp (a traffic light) on a meter panel (not shown) of the vehicle. The ECU 10 controls the display device 20 to display the information on the meter panel.

[0022] The drive unit 30 is a device for applying a driving force to the drive wheels of the host vehicle to make the host vehicle run. The ECU 10 controls the operation of the drive unit 30 to control the driving force applied to the drive wheels.

[0023] The braking device 40 is a device for applying a braking force to the wheels of the host vehicle in order to brake the host vehicle. The ECU 10 controls the operation of the braking device 40 to control the braking force applied to the wheels.

[0024] (Details of operation) Next, the operation of the ECU 10 will be described in detail with reference to Fig. 2. Fig. 2 is a diagram showing a processing procedure in which the CPU of the ECU 10 recognizes the traffic light for the vehicle's own lane and controls the devices 20 to 40. The CPU is configured to repeatedly execute the processing shown in Fig. 2 in this order every time a predetermined time elapses while the ignition switch is in the ON state.

[0025] At a predetermined timing, the CPU starts the process from step 200 and proceeds to step 205 , where it acquires image data from the camera sensor 11 and acquires three-dimensional point cloud information from the Lidar sensor 12 .

[0026] Next, the CPU proceeds to step 210, where it generates a captured image based on the image data acquired in step 205, and extracts a light region from the captured image. The light region is an area where lights (traffic lights, street lights, tail lights of a preceding vehicle, head lights of an oncoming vehicle, etc.) may exist, and is extracted as follows. That is, the CPU generates a captured image as a color image based on the image data, calculates the RGB values ​​(color information) of each pixel constituting the color image, and judges whether the calculated RGB value for each pixel is included in the "range of average RGB values ​​of lights". Here, the "range of average RGB values ​​of lights" is a range determined by experiments or simulations, and is stored in advance in the ROM of the ECU 10. If the RGB value of a certain pixel is included in the above range, the CPU judges the pixel to be a light candidate. If the distance between the pixels judged to be light candidates is equal to or less than a predetermined distance threshold, the CPU extracts a predetermined range including these pixels as the light region. As described below, the light area is used when detecting the traffic light housing, so it is desirable for the above-mentioned "predetermined range" to be a rectangular range that includes the periphery of these pixel groups and the area below them.

[0027] The method of extracting the light area is not limited to the above. For example, the CPU may generate a captured image as a grayscale image based on image data, and calculate the luminance value (brightness information) of each pixel constituting the grayscale image. Then, a predetermined range including a pixel group having a luminance value higher than a predetermined luminance threshold may be extracted as the light area. However, as described later, in this embodiment, the color of the signal light is displayed on the meter panel, so even if the light area is extracted based on the grayscale image, it is preferable that the color information of the signal light is calculated by another method.

[0028] Next, the CPU proceeds to step 215 and determines whether or not a traffic light exists in the light area extracted in step 210 based on the three-dimensional point cloud information acquired in step 205. Specifically, the CPU first selects a spatial coordinate group corresponding to the position coordinates of each pixel constituting the light area from the spatial coordinate group included in the three-dimensional point cloud information (hereinafter, the selected spatial coordinate group is also referred to as a "spatial coordinate group corresponding to the light area"). Here, the ROM (storage device) of the ECU 10 stores in advance housing information including the shapes of the housings of various types of traffic lights (specifically, traffic lights for vehicles) (in other words, the spatial coordinate group of the housings). The CPU calculates the degree of similarity between the "shape of the three-dimensional object included in the spatial coordinate group corresponding to the light area" and the "shape of the traffic light housing included in the housing information". If the degree of approximation is equal to or greater than a predetermined approximation threshold, it is determined that a traffic light is present in the light area (in other words, the lights included in the light area are signal lights), and if the degree of approximation is less than the approximation threshold, it is determined that a traffic light is not present in the light area (in other words, the lights included in the light area are lights other than signal lights). The CPU executes the process of step 215 for all light areas extracted in step 210. If there is a light area where it is determined that a traffic light is present, the CPU proceeds to step 220. On the other hand, if there is no light area where it is determined that a traffic light is present, the CPU proceeds to step 255 and temporarily ends the process shown in FIG. 2.

[0029] In step 220, the CPU extracts the casing of the traffic light (strictly speaking, the group of spatial coordinates representing the casing) from the group of spatial coordinates corresponding to the light area in which it was determined in step 215 that the traffic light exists. The CPU then proceeds to step 225 and determines whether or not the traffic light is a traffic light for the current lane based on the three-dimensional point cloud information and the relative position (distance and orientation) and shape (height and direction) of the traffic light casing extracted in step 220.

[0030] For example, if a solid object (e.g., a median strip) different from other vehicles is detected between the vehicle and the traffic light by the three-dimensional point cloud information, the CPU determines that the traffic light is not a traffic light for the vehicle's own lane (e.g., a traffic light for a cross lane that crosses the vehicle's own lane). Also, if the relative position of the traffic light clearly deviates from the roadway including the vehicle's own lane and a lane traveling in the same direction as the vehicle's own lane, the CPU determines that the traffic light is not a traffic light for the vehicle's own lane (e.g., a traffic light for a cross lane). Furthermore, if the height of the traffic light is equal to or greater than a predetermined height threshold, the CPU determines that the traffic light is not a traffic light for the vehicle's own lane (e.g., a traffic light for a cross lane that crosses the vehicle's own lane in a multi-level intersection). Alternatively, in countries where traffic lights are installed on a lane-by-lane basis, if the lateral position of the traffic light (the position in the vehicle's width direction) is significantly separated from the vehicle, the CPU determines that the traffic light is not a traffic light for the vehicle's own lane (e.g., a traffic light for an adjacent lane). In contrast, for example, if a traffic light is located in front of the vehicle, no three-dimensional object other than another vehicle is detected between the vehicle and the traffic light, and the height of the traffic light is less than the height threshold, the CPU determines that the traffic light is for the vehicle's lane.

[0031] The CPU executes the process of step 225 for all traffic light housings extracted in step 220. If a traffic light for the own lane is present, the CPU proceeds to step 230. On the other hand, if a traffic light for the own lane is not present, the CPU proceeds to step 255 and temporarily ends the process shown in Fig. 2. Note that since the Lidar sensor 12 is capable of relatively long-distance ranging and spatial imaging, there are cases in which multiple traffic lights for the own lane are present (detected).

[0032] In step 230, the CPU tracks whether or not the same traffic light information as that of the traffic light determined to be for the own lane in step 225 is stored in the RAM of the ECU 10. Here, the traffic light information is information including the relative position of the traffic light for the own lane and the color of the signal light. If traffic light information for the same traffic light is stored in the RAM, the CPU updates the information in step 235. Specifically, the relative position of the traffic light is updated and the color of the traffic light is added. By adding the color of the traffic light, the transition of the color of the traffic light is stored (described later). On the other hand, if the traffic light information for the same traffic light is not stored in the RAM, the CPU determines that a new traffic light has been detected, and in step 235 stores (memorizes) the traffic light information for that traffic light in the RAM.

[0033] The CPU then proceeds to step 240 and determines the type of traffic light based on the color transition of the signal light included in the traffic light information updated in step 235. Traffic light types are typically normal traffic lights, flashing traffic lights, and warning traffic lights. For example, if the color of the signal light changes in the order "red", "yellow", and "green", the CPU determines that the traffic light is a normal traffic light. On the other hand, if the color of the signal light is flashing "yellow", the CPU determines that the traffic light is a flashing yellow traffic light.

[0034] Next, the CPU proceeds to step 245 and controls the display device 20 to display information about the color of the signal light on the meter panel. If multiple traffic lights for the current lane are detected in step 225, the CPU may display the information about the color of the signal light in the order in which the traffic lights are encountered, or in association with the distance to the traffic light.

[0035] Next, the CPU proceeds to step 250, and controls the driving device 30 and / or the braking device 40 to accelerate or decelerate the vehicle based on the traffic light information (i.e., the distance to the traffic light and the color of the signal light) of the traffic light for the vehicle's lane (strictly speaking, the traffic light closest to the vehicle) stored in the RAM. Note that the order of the processing of steps 245 and 250 does not matter. After that, the CPU proceeds to step 255 and temporarily ends the processing shown in FIG. 2.

[0036] As described above, the present embodiment determines whether or not a traffic light is present based on the three-dimensional point cloud information acquired by the Lidar sensor 12. The three-dimensional point cloud information can be acquired regardless of the brightness of the surroundings. In addition, the three-dimensional point cloud information is three-dimensional information including the relative position and shape of a solid object. Therefore, compared to a configuration that recognizes traffic lights based on a two-dimensional captured image, the traffic lights can be recognized with high accuracy even in a weak light environment, and the possibility of erroneous detection or erroneous selection of a traffic light can be significantly reduced. With this configuration, it is possible to appropriately recognize the traffic light for the own lane even in a weak light environment.

[0037] In particular, the present embodiment is configured to extract a light area based on image data and determine whether a traffic light exists in the light area, rather than determining whether all three-dimensional objects included in the three-dimensional point cloud information are traffic lights. This is based on the knowledge that a traffic light is a type of light and therefore is highly unlikely to exist in an area other than the light area. With this configuration, the number of three-dimensional objects to be determined as to whether a traffic light exists can be significantly reduced, thereby reducing the processing load without reducing the accuracy of traffic light recognition.

[0038] Furthermore, the present embodiment of the device is configured to control the acceleration and deceleration of the vehicle based on the recognized traffic lights. The distance to the traffic lights can be accurately calculated based on the three-dimensional point cloud information. Therefore, when multiple traffic lights are recognized, the timing at which the vehicle will encounter the traffic lights can be accurately calculated based on the distance to the traffic lights. Therefore, the control timing and control amount of the acceleration and deceleration control can be appropriately calculated based on the distance to the traffic lights and the color of the signal lights.

[0039] The above describes the traffic light recognition device according to this embodiment, but the present invention is not limited to the above embodiment, and various modifications are possible without departing from the object of the present invention.

[0040] For example, the traffic light recognition device may be configured not to include the display device 20, the drive device 30, and the braking device 40. In this case, the ECU 10 may be connected to ECUs capable of controlling the display device 20, the drive device 30, and the braking device 40, respectively, and may be configured to be capable of transmitting the calculation results of the ECU 10 to each ECU. [Explanation of symbols]

[0041] 10: traffic light recognition ECU, 11: camera sensor, 12: Lidar sensor, 20: display device, 30: drive device, 40: braking device

Claims

1. It is mounted on the vehicle, an imaging device capable of acquiring image data by imaging an area in front of the vehicle; a light irradiation device that is capable of irradiating light onto a predetermined range including the front area of ​​the vehicle and receiving reflected waves from a three-dimensional object present within the predetermined range to acquire three-dimensional object information including a relative position and a shape of the three-dimensional object; Extracting an area where a light may exist as a light area based on the image data; determining whether or not a traffic light is present in the light area based on the three-dimensional object information; when it is determined that a traffic light is present in the light area, it is determined whether or not the traffic light is a traffic light for the own lane based on the three-dimensional object information. A control unit configured as described above. Equipped with The control unit generating a captured image as a color image based on the image data; Calculating the RGB values ​​of each pixel constituting the color image; For each pixel, determining whether the calculated RGB value falls within a predetermined range; If the distance between pixels whose RGB values ​​are determined to be within the range is equal to or less than a predetermined distance threshold, a rectangular area including the periphery of these pixels and an area below the pixels is extracted as the light area. Traffic light recognition device.

2. The traffic light recognition device according to claim 1, Further comprising a storage device that stores in advance housing information including the shapes of housings of various types of traffic lights, The control unit calculating an approximation of a three-dimensional object included in the three-dimensional object information in an area corresponding to the light area to a casing of a traffic light based on the casing information, and determining that a traffic light exists in the light area when the approximation is equal to or greater than a predetermined approximation threshold value; It was configured as follows: Traffic light recognition device.

3. In the traffic light recognition device according to claim 1 or 2, The light irradiation device is a three-dimensional Lidar. Traffic light recognition device.

Images