Signal recognition device

Abstract

To provide a traffic signal recognition device capable of more accurately recognizing a direction indicated by an arrow even including a traffic signal positioned away from a vehicle under a dark environment such as e.g., at night.SOLUTION: A traffic signal recognition device 100 comprises: a turned-on state recognition section 14 for recognizing a turned-on state of a lamp portion of a traffic signal around a vehicle based on a picked-up image of a camera 2; and an arrow direction recognition section 15 for recognizing a direction indicated by a blue lamp arrow in the traffic signal based on the picked-up image and the turned-on state. In a case where both a first lamp portion corresponding to a lamp other than the arrow and a second lamp portion corresponding to the blue lamp arrow are turned on, the turned-on state recognition section 14 recognizes a turned-on state where the first lamp portion and the second lamp portion are simultaneously turned on. In a case where the simultaneous turning-on is recognized, the arrow direction recognition section 15 recognizes the direction indicated by the blue lamp arrow based on a result of comparing luminance distribution of a first flare generated in the picked-up image with the first lamp portion defined as a light source and luminance distribution of a second flare generated in the picked-up image with the second lamp portion defined as a light source.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a traffic light recognition device. [Background technology]

[0002] A signal recognition system described in Patent Document 1, for example, is known in the past. In the signal recognition system described in Patent Document 1, signal detection information of a target traffic light around a vehicle is acquired based on camera image information from a camera mounted on the vehicle. Light pattern information is acquired that indicates the relative positional relationship between multiple light portions of the traffic light and the appearance of each of the multiple light portions when lit. The light detection information is compared with the light pattern information to recognize the lighting state of the target traffic light. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-2275 Summary of the Invention [Problem to be solved by the invention]

[0004] Unlike lights other than arrows (e.g., red, blue, or yellow circular lights), blue light arrows can be recognized not only based on whether the light is on, but also based on the direction the arrow is pointing. In the technical fields of driving assistance and automated driving systems related to advanced safety technologies, it is desirable to be able to more accurately recognize the direction the arrow is pointing, including traffic lights located far away from the vehicle in dark environments such as at night. [Means for solving the problem]

[0005] A traffic light recognition device according to one embodiment of the present invention includes a lighting state recognition unit that recognizes the lighting state of the light portions of traffic lights around a vehicle based on an image captured by an on-board camera, and an arrow direction recognition unit that recognizes the direction indicated by a blue light arrow at the traffic light based on the captured image and the lighting state. When both a first light portion corresponding to a light other than the arrow and a second light portion corresponding to the blue light arrow are lit, the lighting state recognition unit recognizes the lighting state of the first light portion and the second light portion being simultaneously lit, and when simultaneous lighting is recognized by the lighting state recognition unit, the arrow direction recognition unit recognizes the direction indicated by the blue light arrow based on a comparison result between the luminance distribution of a first flare generated in the captured image using the first light portion as a light source and the luminance distribution of a second flare generated in the captured image using the second light portion as a light source.

[0006] In the traffic light recognition device according to one aspect of the present invention, when the lighting state recognition unit recognizes simultaneous lighting, for example, in a dark environment such as at night, there is a high possibility that a first flare and a second flare are generated in the captured image. Since the first flare occurs in the captured image with the first light portion corresponding to the light other than the arrow as a light source, the luminance distribution on the captured image is concentric with the first light portion. Since the second flare occurs in the captured image with the second light portion corresponding to the blue light arrow as a light source, the luminance distribution on the captured image is biased with respect to the concentric circle corresponding to the second light portion. Therefore, when the lighting state recognition unit recognizes simultaneous lighting, the direction indicated by the blue light arrow can be more accurately recognized based on the bias in the luminance distribution of the second flare based on the luminance distribution of the first flare. This makes it possible to more accurately recognize the direction indicated by the arrow, including traffic lights located far from the vehicle in a dark environment such as at night.

[0007] In one embodiment, the arrow direction recognition unit may not recognize the direction of the blue light arrow when it is determined that blur due to vehicle shaking occurs based on the captured image and the detection result of the vehicle acceleration sensor. In this case, it is possible to suppress a decrease in the accuracy of recognition of the direction of the arrow when motion blur due to vehicle shaking occurs in the captured image. Effect of the Invention

[0008] According to one aspect of the traffic light recognition device of the present invention, it is possible to more accurately recognize the direction indicated by an arrow, including traffic lights located far away from the vehicle in dark environments such as at night. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of a traffic light recognition device according to an embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of a situation in which a candidate area is estimated. [Diagram 3] FIG. 11 is a diagram showing another example of a situation in which a candidate area is estimated. [Figure 4] FIG. 4 is a diagram showing an example of a lighting state of a first lighting portion. [Diagram 5] FIG. 4 is a diagram showing an example of the luminance distribution of a first lamp portion. [Figure 6] FIG. 4 is a diagram showing an example of a state in which first and second lamp portions are simultaneously turned on. [Figure 7] FIG. 4 is a diagram showing an example of the luminance distribution of first and second lamp portions. [Figure 8] 2 is a flowchart showing an example of processing performed by the traffic light recognition device of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same or corresponding elements are designated by the same reference numerals, and duplicated description will be omitted.

[0011] Fig. 1 is a block diagram showing a schematic configuration of a traffic light recognition device according to an embodiment. The traffic light recognition device 100 shown in Fig. 1 is mounted on a vehicle such as a passenger car, and recognizes the illumination state of a traffic light included in an image captured by a camera of the vehicle.

[0012] As shown in Fig. 1, the traffic light recognition device 100 includes an ECU (Electronic Control Unit) 10 that performs overall control of the device. The ECU 10 is an electronic control unit that includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The ECU 10 loads programs stored in the ROM into the RAM and executes them with the CPU to perform various vehicle controls. The ECU 10 may be composed of multiple electronic control units.

[0013] The ECU 10 is connected to a GPS receiving unit 1, a camera (vehicle-mounted camera) 2, a radar sensor 3, an acceleration sensor 4, and a map database 5.

[0014] The GPS receiver 1 measures the position of the vehicle (for example, the latitude and longitude of the vehicle) by receiving signals from three or more GPS satellites. The GPS receiver 1 transmits the measured position information of the vehicle to the ECU 10.

[0015] The camera 2 is an imaging device that captures an image of the external situation of the vehicle. As an example, the camera 2 is provided behind the windshield of the vehicle and captures an image of the area in front of the vehicle. The camera 2 transmits the captured image of the area in front of the vehicle to the ECU 10. The camera 2 may be a monocular camera or a stereo camera. The mounting location of the camera 2 is not limited to the back side of the windshield, and it is sufficient if the camera can capture at least an image of the area in front of the vehicle. The camera 2 is not limited to the area in front of the vehicle, and may be composed of multiple cameras that capture images in multiple directions of the vehicle.

[0016] The radar sensor 3 is a detection device that detects objects (electric poles, traffic lights, etc.) around the vehicle using radio waves (e.g., millimeter waves) or light. The radar sensor 3 includes, for example, a millimeter wave radar or a LiDAR (Light Detection and Ranging). The radar sensor 3 detects objects by transmitting radio waves or light to the vicinity of the vehicle and receiving the radio waves or light reflected by the objects. The radar sensor 3 transmits detection results related to the detected objects to the ECU 10.

[0017] The acceleration sensor 4 is a detector that detects the acceleration of the vehicle. The acceleration sensor 4 includes, for example, a vertical acceleration sensor that detects vertical acceleration, which is the acceleration in the up and down direction of the vehicle, and a lateral acceleration sensor that detects lateral acceleration, which is the acceleration in the left and right direction (horizontal direction) of the vehicle. The acceleration sensor 4 may also include a vertical acceleration sensor that detects vertical acceleration, which is the acceleration in the front-rear direction (vertical direction) of the vehicle. The acceleration sensor 4 transmits vehicle acceleration information to the ECU 10.

[0018] The map database 5 is a database that stores map information. The map database 5 is formed, for example, in an HDD [Hard Disk Drive] mounted on the vehicle. The map information includes road position information, road shape information (e.g., curves, types of straight sections, curvature of curves, etc.), intersection and branch point position information, and traffic light position information on the map. The traffic light position information on the map is information on the position coordinates of the traffic light on the map. The map information may include surrounding structure data associated with the traffic light position information on the map. Note that the map database 5 does not necessarily need to be mounted on the vehicle, and may be formed on a server that can communicate with the vehicle.

[0019] The surrounding structure data includes position information on the map of surrounding structures that exist around the traffic light and shape information of the surrounding structures. The surrounding structures are structures that exist around the traffic light and are likely to have the traffic light installed therein. Examples of the surrounding structures may include support pillars, utility poles, poles, pedestrian bridges, and inner walls of tunnels. The surrounding structures do not necessarily need to include the entire structure, and may include only a portion of the structure within a certain range from the traffic light. The position information of the surrounding structures on the map is information on the three-dimensional position of the surrounding structures on the map. The shape information of the surrounding structures is information on the three-dimensional shape of the surrounding structures.

[0020] Next, a description will be given of the functional configuration of the ECU 10. The ECU 10 has a vehicle position recognition unit 11, an object recognition unit 12, a candidate area estimation unit 13, a lighting state recognition unit 14, and an arrow direction recognition unit 15. Note that some of the functions of the ECU 10 described below may be executed by a server capable of communicating with the vehicle.

[0021] The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the position information of the GPS receiver 1 and the map information of the map database 5. The vehicle position recognition unit 11 may also recognize the position of the vehicle on the map by SLAM [Simultaneous Localization and Mapping] technology using position information of fixed obstacles such as utility poles included in the map information of the map database 5 and the captured image of the camera 2 or the detection result of the radar sensor 3. The vehicle position recognition unit 11 may use traffic light data and surrounding structure data stored in the map database 5 in addition to the map information of the map database 5. The vehicle position recognition unit 11 may also recognize the position of the vehicle on the map by other well-known methods.

[0022] The object recognition unit 12 acquires an image of the area in front of the vehicle, including color information, based on an image captured by the camera 2. The object recognition unit 12 recognizes objects around the vehicle, for example, based on the image captured by the camera 2 and the detection result of the radar sensor 3. The object recognition unit 12 may perform object recognition by integrating the image captured by the camera 2 and the detection result of the radar sensor 3 using a well-known sensor information integration process. The object recognition unit 12 performs sensor information integration, for example, by adding color information captured by the camera 2 to three-dimensional detection points (reflection points of light or millimeter waves) detected by the radar sensor 3. The object recognition unit 12 recognizes the relative position of the object with respect to the vehicle and the shape of the object.

[0023] Note that the object recognition unit 12 does not necessarily need to use both the captured image of the camera 2 and the detection result of the radar sensor 3. The object recognition unit 12 may use only the captured image of the camera 2 for object recognition. When only the captured image of the camera 2 is used for object recognition, it is not necessary to include the radar sensor 3.

[0024] The candidate area estimation unit 13 estimates a candidate area based on the position of the vehicle on the map recognized by the vehicle position recognition unit 11, the recognition result of the object recognition unit 12, and the map information of the map database 5. The candidate area is an area in the captured image that is a candidate for a traffic light that faces the vehicle ahead in the traveling direction of the vehicle. The candidate area may be a search area [ROI: Region of Interest] set in the captured image. The search area is an area in the captured image where image recognition processing is performed to recognize the lighting state of the traffic light. By using the candidate area, it is possible to identify the traffic light that is the target of the lighting state recognition even in a situation where a light source other than the traffic light is present around the vehicle in a dark environment such as at night.

[0025] FIG. 2 is a diagram showing an example of a situation in which a candidate area is estimated. FIG. 2 is a captured image captured by a camera of a vehicle traveling on a road with one lane in each direction and a crossroad with traffic lights ahead. FIG. 3 is a diagram showing another example of a situation in which a candidate area is estimated. FIG. 3 is a captured image captured by a camera of a vehicle traveling on a road with one lane in each direction and a preceding vehicle that stops in front of a crosswalk with traffic lights ahead. The candidate area estimation unit 13 may, for example, use a machine learning result of an image including a traffic light to estimate a portion in the captured image that includes an object similar to the outline of a traffic light as a candidate area (area A1 in FIG. 2). For example, when the candidate area estimation unit 13 recognizes a group of red lights specific to a stopped preceding vehicle in the captured image, it may estimate a portion in the captured image that includes the preceding vehicle and the area above it as a candidate area (area A2 in FIG. 3). For example, the candidate area estimation unit 13 may estimate a portion in the captured image that includes surrounding structures as a candidate area (area A3 in FIG. 2 and FIG. 3). The candidate area estimation unit 13 may, for example, recognize a portion in the captured image that includes a pair of traffic lights facing each other in a direction that intersects with the traveling direction of the vehicle (area A4 in FIG. 2), and estimate a portion that includes a traffic light between the pair of traffic lights as a candidate area (area A1 in FIG. 2). The candidate area estimation unit 13 may, for example, recognize a portion in the captured image that corresponds to an intersection located near the vehicle position recognized by the vehicle position recognition unit 11 (for example, an intersection when the vehicle position reaches a white line and a sign a predetermined distance before the intersection ahead in the traveling direction of the vehicle) as a candidate area (area A5 in FIG. 2).

[0026] The candidate area estimation unit 13 recognizes the candidate area so as to include the position of a traffic light facing the vehicle in the captured image. In the examples of Fig. 2 and Fig. 3, the candidate area is set as a rectangular region. Note that the shape of the candidate area is not limited to a rectangular shape, and may be an ellipse, a circle, a polygon, or other shapes. Furthermore, the method of estimating the candidate area is not limited to the above method.

[0027] The lighting state recognition unit 14 recognizes the lighting state (described later) of the light parts of the traffic light in front of (around) the vehicle based on the image captured by the vehicle-mounted camera. Generally, a traffic light has multiple light parts. A light part is a part that is turned on and off. The light part is composed of, for example, a light bulb, an LED, a light-emitting device, a display, etc.

[0028] When a candidate area exists in the captured image, the lighting state recognition unit 14 recognizes the lighting state of the traffic light in the candidate area. When a candidate area estimated by the candidate area estimation unit 13 (for example, at least one of the examples in Figs. 2 and 3 described above) exists in the captured image, the lighting state recognition unit 14 recognizes the position of the traffic light in the captured image of the camera 2 based on the result of matching the traffic light and surrounding structures extracted from the map database 5 with the recognition result of the object recognition unit 12. The lighting state recognition unit 14 can recognize the position of the traffic light in the captured image by a well-known method. The lighting state recognition unit 14 recognizes the lighting state of the traffic light by performing image recognition (for example, pattern matching considering color) in the candidate area. The processing content of the image recognition is not particularly limited, and it may be processing capable of recognizing the lighting state of the traffic light.

[0029] The lighting state of a traffic signal includes a state in which a first light portion corresponding to a light other than an arrow is on. The first light portion is typically a circular signal, and is used as a main light source of, for example, a yellow light or a red light. FIG. 4 is a diagram showing an example of a lighting state of the first light portion. In FIG. 4, for example, three examples (40 m, 80 m, and 120 m) of different distances from the vehicle to the signal are shown as captured images showing a lighting state in which a yellow light is on by itself. The upper part of FIG. 4 is a life-size captured image, and the lower part is an enlarged view of the first light portion in the upper part.

[0030] As shown in FIG. 4, when the yellow light is turned on, a high-luminance portion corresponding to the light source portion and a flare portion spreading around the light source portion appear on the captured image. Flare is a light fogging phenomenon that occurs when noise light is reflected on the lens surface or the like by pointing the lens of the camera 2 at the light portion of the traffic light. In the flare portion, a part of the image becomes whitish, and sharpness and contrast decrease. Flare is likely to occur, for example, in images captured at night. And, the greater the distance from the vehicle to the traffic light, the more the image of the high-luminance portion of the yellow light in the captured image tends to be crushed, and the boundary between the high-luminance portion and the flare portion tends to become blurred.

[0031] Fig. 5 is a diagram showing an example of the luminance distribution of the first light portion. For convenience of explanation, Fig. 5 is a grayscale diagram in which the color information of the enlarged view of 80 m in Fig. 4 is inverted. As shown in Fig. 5, a two-dot chain line LS and a dashed line FL are drawn.

[0032] The two-dot chain line LS defines a circular area corresponding to the area where the light source of the first light portion exists. The two-dot chain line LS may be obtained as a high-luminance area when the luminance distribution of the captured image is binarized with a predetermined light source threshold value, for example. The shape of the two-dot chain line LS may be used to determine the direction of the arrow of the second light portion by comparing the shapes of the second light portion and the light source described below.

[0033] The dashed line FL defines an area corresponding to a first flare occurring in a captured image with the first light portion as a light source. In the example of FIG. 5, a flare occurs isotropically from a circular light source, and the dashed line FL circle is substantially concentric with the two-dot chain line LS circle and has a larger radius than the two-dot chain line LS circle. The dashed line FL may be acquired as a medium luminance portion when the luminance distribution of the captured image is binarized with a predetermined flare threshold value. The flare threshold value is a threshold value having a lower luminance value than the light source threshold value. The shape of the dashed line FL may be used to determine the direction indicated by the arrow of the second light portion by comparing the spread of the flare of the second light portion described later.

[0034] The lighted state of the traffic light includes a state in which the second light portion corresponding to the blue light arrow is on. The second light portion is typically an arrow signal, and is an auxiliary light source that supplements, for example, a red light. The second light portion has a light source area smaller than that of the first light portion. The direction in which the arrow of the second light portion points is not particularly limited. The lighted state of the traffic light also includes a simultaneous lighted state in which both the first light portion and the second light portion are on. Simultaneous lighted here means a state in which both the red light and the green arrow signal are on.

[0035] The lighting state recognition unit 14 recognizes the lighting state of the first lighting portion and the second lighting portion simultaneously when both the first lighting portion corresponding to the light other than the arrow and the second lighting portion corresponding to the blue light arrow are turned on. The lighting state recognition unit 14 does not recognize the lighting state of the first lighting portion and the second lighting portion simultaneously when either the first lighting portion corresponding to the light other than the arrow and the second lighting portion corresponding to the blue light arrow is not turned on.

[0036] Fig. 6 is a diagram showing an example of a simultaneous lighting state of the first and second light parts. In Fig. 6, three examples (40m, 80m, and 120m) of different distances from the vehicle to the signal are shown as captured images showing a simultaneous lighting state in which, for example, a red light and a green arrow signal are simultaneously turned on. The upper part of Fig. 6 is a life-size captured image, and the lower part is an enlarged view of the first and second light parts of the upper part.

[0037] As shown in Fig. 6, when the red light and the green arrow signal are lit at the same time, a high-luminance part corresponding to the light source part and a flare part spreading around the light source part appear in the captured image of the red light and the blue arrow signal, respectively. As in the example of Fig. 4, the greater the distance from the vehicle to the traffic light, the more the image of the high-luminance part of the red light and the blue arrow signal in the captured image tends to be blurred, and the boundary between the high-luminance part and the flare part tends to become blurred. Therefore, for the blue arrow signal, the greater the distance from the vehicle to the traffic light, the more difficult it becomes to recognize the direction it is pointing.

[0038] The arrow direction recognition unit 15 recognizes the direction of the blue light arrow on the traffic light based on the captured image and the lighting state. For example, when simultaneous lighting is recognized by the lighting state recognition unit 14, the arrow direction recognition unit 15 recognizes the direction of the blue light arrow based on a comparison result between the luminance distribution of the first flare generated in the captured image with the first light portion as a light source and the luminance distribution of the second flare generated in the captured image with the second light portion as a light source. The comparison between the luminance distribution of the first flare and the luminance distribution of the second flare means, for example, whether or not the luminance distribution with the blue light arrow as a light source on the captured image is biased with respect to the luminance distribution with lights other than the arrow as a light source.

[0039] Fig. 7 is a diagram showing an example of the luminance distribution of the first and second light portions. For convenience of explanation, Fig. 7 shows a grayscale image obtained by inverting the color information of the 80m enlarged view of Fig. 6. Note that when recognizing that the direction is the direction indicated by the blue arrow signal, recognition including color information may be performed.

[0040] As shown in Fig. 7, dashed lines FL1 and FL2 are drawn. The dashed line FL1 defines an area corresponding to a first flare occurring in an image captured with the red light in Fig. 6 as a light source. In the red light in Fig. 6, flare occurs isotropically from a circular light source, so the circle of the dashed line FL1 can be equivalent to the circle of the yellow light dashed line FL in Fig. 5. The luminance distribution of the first flare is approximately uniform within the circle of the dashed line FL1.

[0041] The dashed line FL2 is illustrated for convenience in order to compare the luminance distribution of the second flare with that of the first flare. The dashed line FL2 is a circular region including the second flare that occurs in the captured image with the blue arrow signal in FIG. 6 as a light source, and is shown in the same shape as the dashed line FL1 as an example. In the blue arrow signal in FIG. 6, the flare occurs from an arrow-shaped light source with a smaller light-emitting area than the circular red signal, so the luminance distribution of the second flare is not uniform in the dashed line FL2, and the luminance is higher on the tip side (pointing direction) of the blue arrow and lower on the base side (opposite to the pointing direction) of the blue arrow. In the example of FIG. 7, the luminance distribution of the second flare has dark parts DZ1 and DZ2 that are low in the upper left part and the lower left part within the circle of the dashed line FL2. In this case, the arrow direction recognition unit 15 can recognize that the right side of the paper surface of FIG. 7, which is opposite to the dark parts DZ1 and DZ2 with respect to the second flare, is the direction pointed by the blue arrow signal.

[0042] Furthermore, arrow direction recognition unit 15 may calculate, for example, a luminance histogram in the up-down and left-right directions for each pixel in a portion of the luminance distribution of an image captured using the red signal and blue arrow signal in FIG. 6 as light sources where the luminance is equal to or greater than the above-mentioned flare threshold, and recognize that the direction is that indicated by the blue arrow signal based on the direction in which the peak position on the luminance histogram of the blue arrow signal is biased relative to the peak position on the luminance histogram of the red signal.

[0043] Alternatively, the arrow direction recognition unit 15 may recognize the dark portions DZ1 and DZ2 based on whether the color of pixels in a captured image using the red signal and the blue arrow signal in Fig. 6 as light sources is red or blue (or green). When the arrow direction recognition unit 15 recognizes the dark portions DZ1 and DZ2, it may recognize high-luminance portions adjacent to the dark portions DZ1 and DZ2 in the captured image using a predetermined luminance threshold, and recognize the opposite side of the high-luminance portions to the dark portions DZ1 and DZ2 as the direction indicated by the blue arrow signal.

[0044] In addition, for example, when the distance is 40 m in Fig. 6, the shape of the high-luminance part corresponding to the light source part is such that the direction indicated by the blue arrow signal can be recognized, compared with the case where the distance is 80 m or more. In this case, the arrow direction recognition unit 15 may obtain an area along the shape of the light source itself, such as the two-dot chain line LS in Fig. 5, for each light source of the red light and the blue arrow signal, and recognize the direction indicated by the blue light arrow by comparing the shapes.

[0045] Incidentally, when it is determined that blurring due to vehicle shaking has occurred based on the captured image and the detection result of the acceleration sensor 4 of the vehicle, the arrow direction recognition unit 15 does not need to recognize the direction indicated by the blue light arrow. For example, based on the captured image, the arrow direction recognition unit 15 may determine that both the first blur and the second blur are caused by vehicle shaking when a first blur occurring in the captured image with the first light portion as a light source and a second blur occurring in the captured image with the second light portion as a light source extend in the vehicle up-down direction in the captured image with an equal amount of blur. Based on the detection result of the acceleration sensor of the vehicle, the arrow direction recognition unit 15 may determine that both the first blur and the second blur are caused by vehicle shaking when the vertical acceleration of the vehicle in the up-down direction is equal to or greater than a predetermined vertical acceleration threshold, or when the lateral acceleration of the vehicle in the lateral direction is equal to or greater than a predetermined lateral acceleration threshold.

[0046] Next, an example of traffic light recognition processing by the traffic light recognition device 100 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of processing by the traffic light recognition device of Fig. 1. The processing of the flowchart shown in Fig. 8 is repeatedly performed at a predetermined cycle, for example, while the vehicle is traveling.

[0047] 8, in step S01, the ECU 10 recognizes the vehicle position by the vehicle position recognition unit 11. The vehicle position recognition unit 11 recognizes the position of the vehicle on a map based on, for example, the position information of the GPS receiver 1 and the map information of the map database 5.

[0048] In step S02, the ECU 10 recognizes objects using the object recognition unit 12. The object recognition unit 12 acquires an image of the area in front of the vehicle, including color information, based on an image captured by the camera 2. The object recognition unit 12 recognizes objects around the vehicle, for example, based on the image captured by the camera 2 and the detection result of the radar sensor 3.

[0049] In step S03, the ECU 10 estimates a candidate area using the candidate area estimation unit 13. The candidate area estimation unit 13 estimates a candidate area based on, for example, the position of the vehicle on the map recognized by the vehicle position recognition unit 11, the recognition result by the object recognition unit 12, and the map information in the map database 5.

[0050] In step S04, the ECU 10 determines whether or not there is a candidate area in the captured image by the lighting state recognizing unit 14. When a candidate area is estimated in the captured image by the candidate area estimating unit 13, the lighting state recognizing unit 14 determines that there is a candidate area in the captured image. When the ECU 10 determines that there is a candidate area in the captured image (step S04: YES), the ECU 10 proceeds to step S05. On the other hand, when the ECU 10 determines that there is no candidate area in the captured image (step S04: NO), the ECU 10 ends the current process of FIG. 8.

[0051] In step S05, the ECU 10 recognizes the lighting state by the lighting state recognition unit 14. The lighting state recognition unit 14 recognizes the lighting state of the lights of the traffic lights in front of (around) the vehicle based on an image captured by an on-board camera. The lighting state recognition unit 14 recognizes the lighting state of the traffic lights in the candidate area, for example, by performing image recognition within the candidate area (for example, pattern matching taking color into consideration).

[0052] In step S06, the ECU 10 judges whether or not the lighting state with simultaneous lighting has been recognized by the lighting state recognition unit 14. When both the first light portion corresponding to the light other than the arrow and the second light portion corresponding to the blue light arrow are on, the lighting state recognition unit 14 judges that the lighting state with simultaneous lighting of the first light portion and the second light portion has been recognized. When either the first light portion corresponding to the light other than the arrow or the second light portion corresponding to the blue light arrow is not on, the lighting state recognition unit 14 judges that the lighting state with simultaneous lighting has not been recognized. When the ECU 10 judges that the lighting state with simultaneous lighting has been recognized (step S06: YES), it proceeds to step S07. On the other hand, when the ECU 10 judges that the lighting state with simultaneous lighting has not been recognized (step S06: NO), it ends the current process of FIG. 8.

[0053] In step S07, the ECU 10 determines whether or not blur (first blur and second blur) caused by vehicle shaking has occurred by the arrow direction recognition unit 15. The arrow direction recognition unit 15 may determine that the first blur and the second blur caused by vehicle shaking have occurred when the first blur caused in the captured image with the first lamp portion as a light source and the second blur caused in the captured image with the second lamp portion as a light source extend in the vehicle up-down direction in the captured image with an equal blur amount based on the captured image. Alternatively, the arrow direction recognition unit 15 may determine that the first blur and the second blur caused by vehicle shaking have occurred based on the detection result of the acceleration sensor 4 of the vehicle when the vertical acceleration of the vehicle in the up-down direction is equal to or greater than a predetermined vertical acceleration threshold, or when the horizontal acceleration of the vehicle in the left-right direction is equal to or greater than a predetermined horizontal acceleration threshold. When the ECU 10 determines that blur caused by vehicle shaking has not occurred (step S07: YES), the process proceeds to step S08. On the other hand, when the ECU 10 determines that blurring due to vehicle shaking is occurring (step S07: NO), the ECU 10 ends the current process of FIG.

[0054] In step S08, ECU 10 recognizes the arrow direction by arrow direction recognition unit 15. When simultaneous illumination is recognized by illumination state recognition unit 14, arrow direction recognition unit 15 recognizes the direction indicated by the blue light arrow based on a comparison result between the luminance distribution of the first flare occurring in the captured image with the first light portion as a light source and the luminance distribution of the second flare occurring in the captured image with the second light portion as a light source. After that, ECU 10 ends the current process of FIG. 8.

[0055] As described above, when the lighting state recognition unit 14 recognizes simultaneous lighting, for example, in a dark environment such as at night, there is a high possibility that the first flare and the second flare are generated in the captured image. Since the first flare occurs in the captured image with the first light portion corresponding to the light other than the arrow as a light source, the luminance distribution on the captured image is a concentric circle corresponding to the first light portion. Since the second flare occurs in the captured image with the second light portion corresponding to the blue light arrow as a light source, the luminance distribution on the captured image is biased with respect to the concentric circle corresponding to the second light portion. Therefore, according to the traffic light recognition device 100 as described above, when the lighting state recognition unit 14 recognizes simultaneous lighting, the arrow direction recognition unit 15 can more accurately recognize the direction indicated by the blue light arrow based on the bias of the luminance distribution of the second flare based on the luminance distribution of the first flare. This makes it possible to more accurately recognize the direction indicated by the arrow, including traffic lights located far from the vehicle in a dark environment such as at night.

[0056] In the traffic light recognition device 100, the arrow direction recognition unit 15 does not recognize the direction of the blue light arrow when it is determined that blurring due to vehicle shaking occurs based on the captured image and the detection result of the vehicle acceleration sensor 4. This makes it possible to suppress a decrease in the accuracy of recognition of the arrow direction in a situation where motion blurring due to vehicle shaking occurs in the captured image.

[0057] [Variations] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment. The present invention can be embodied in various forms including the above-mentioned embodiment and various modifications and improvements based on the knowledge of those skilled in the art.

[0058] In the above-described traffic light recognition device 100, the arrow direction recognition unit 15 is configured not to recognize the direction indicated by the blue light arrow when it is determined that blurring due to vehicle swaying is occurring, but this process (step S07 in FIG. 8) may be omitted.

[0059] In the above embodiment, the first light portion corresponding to the light other than the arrow is a red light in the simultaneous lighting state, but the present invention is not limited to this. The first light portion that is simultaneously lit with the second light portion (blue light arrow) may be a blue or yellow light other than red. For example, for a moving object such as a tram, whose progress is determined by a light arrow of a color other than blue (e.g., yellow), the present invention can be applied by replacing the second light portion with a third light portion corresponding to a light arrow of a color other than blue.

[0060] In the above embodiment, the traffic light in front of the vehicle has been described, but the traffic light around the vehicle may be the subject of imaging. Also, the radar sensor 3 is not necessarily provided. For example, instead of the radar sensor 3, a TOF [Time Of Flight] sensor, a far infrared ray camera, or the like may be used. [Explanation of symbols]

[0061] 2... camera (vehicle-mounted camera), 4... acceleration sensor, 14... lighting state recognition unit, 15... arrow direction recognition unit, 100... traffic light recognition device.

Claims

1. a lighting state recognition unit that recognizes lighting states of lights of traffic lights around the vehicle based on an image captured by an in-vehicle camera; and an arrow direction recognition unit that recognizes a direction indicated by a blue light arrow at the traffic light based on the captured image and the lighting state, The lighting state recognition unit recognizes the lighting state of the first lighting portion and the second lighting portion when both the first lighting portion corresponding to the light other than the arrow and the second lighting portion corresponding to the blue light arrow are turned on, When the simultaneous lighting is recognized by the lighting state recognition unit, the arrow direction recognition unit recognizes the direction indicated by the blue light arrow based on a comparison result between a luminance distribution of a first flare generated in the captured image using the first light portion as a light source and a luminance distribution of a second flare generated in the captured image using the second light portion as a light source.

2. 2. The traffic light recognition device according to claim 1, wherein the arrow direction recognition unit does not recognize the direction indicated by the blue light arrow when it determines, based on the captured image and the detection results of the vehicle's acceleration sensor, that blurring due to the swaying of the vehicle is occurring.

Images