Signal Information Determination Device

Abstract

To provide a traffic signal information determining device capable of determining display of a traffic signal even when an image in front of a vehicle is obstructed.SOLUTION: A traffic signal information determining device mounted on a vehicle (100) comprises: an imaging unit that acquires an image; and a controller for determining display of a target traffic signal (F1) as a target traffic signal (F1) that displays signal information of a driving lane of the vehicle (100). The imaging unit acquires an image of a reflecting object (w1) positioned around the vehicle (100) and the controller determines display of the target traffic signal (F1) based on a reflected image (Fa) of the target traffic signal (F1) included in an image (v1) of the reflecting object.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a traffic light information determination device. [Background technology]

[0002] Patent Document 1 describes a vehicle driving assistance device that extracts traffic lights that are red from an image of the area ahead of the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-170154 Summary of the Invention [Problem to be solved by the invention]

[0004] If there is an obstacle such as a truck between the traffic light and the vehicle, or if the sun is behind the traffic light, it becomes difficult to determine the traffic light display from the image in front of the vehicle.

[0005] An object of the present invention is to provide a traffic light information determination device that can determine the display of a traffic light even when an image ahead of the vehicle is obstructed. [Means for solving the problem]

[0006] (1) One aspect of the present invention is A traffic light information determination device mounted on a vehicle, a photographing unit that acquires video; a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and Equipped with The imaging unit is An image of the front of the vehicle; Images of reflective objects located around the vehicle and Get The controller When a normal judgment result of the display of the target traffic light cannot be obtained based on the image of the front, The display of the target traffic light is determined based on the reflected image of the target traffic light included in the image of the reflecting object. (2) Another aspect of the present invention is a method for producing a A traffic light information determination device mounted on a vehicle, a photographing unit that acquires video; a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and Equipped with The photographing unit acquires an image of a reflecting object located around the vehicle, The controller estimates the reflection position of the target traffic light based on the position of the target traffic light and the position of the reflecting object, and determines the display of the target traffic light based on the reflected image of the target traffic light included in the image of the reflection position of the reflecting object. (3) Another aspect of the present invention is a method for producing a A traffic light information determination device mounted on a vehicle, a photographing unit that acquires video; a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and Equipped with The photographing unit acquires an image of a reflecting object located around the vehicle, The controller determining the display of the target traffic signal based on a reflected image of the target traffic signal included in the image of the reflecting object; Furthermore, a validation process is performed to determine whether the reflected image represents the target traffic light; The validation process includes: A process of predicting a distorted shape of the target traffic light due to reflection; comparing the reflected image with the predicted distorted shape; The present invention is characterized by comprising: [Effects of the Invention]

[0007] According to the present invention, the controller can determine the display of the target traffic signal based on the reflected image of the target traffic signal reflected by reflecting objects around the vehicle. Therefore, even if the target traffic signal is located in front of the vehicle and obstructs the image in front of the vehicle, it is possible to determine the display of the target traffic signal. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a traffic light information determination device and a vehicle according to an embodiment of the present invention; [Figure 2]FIG. 10 is a diagram illustrating a process for acquiring the position of a target traffic light. [Figure 3] 10A and 10B are diagrams illustrating a process for acquiring the position of a reflecting object. [Figure 4] 10A and 10B are diagrams illustrating a process of estimating the reflection position of a target traffic light. [Figure 5] 10A and 10B are diagrams illustrating a signal display determination process and a validity confirmation process. [Figure 6] 4 is a flowchart illustrating an example of a control process of a traffic light information determination device. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described in detail with reference to the accompanying drawings. Fig. 1 is a block diagram showing a traffic light information determination device and a vehicle according to an embodiment of the present invention.

[0010] The traffic light information determination device 1 of this embodiment is a device that is mounted on a vehicle 100 and can determine the display of a target traffic light even when an image ahead of the vehicle 100 is obstructed. The target traffic light means a traffic light that displays signal information for the lane in which the vehicle 100 is traveling.

[0011] The vehicle 100 includes wheels 102 and 103, a power source 104 that generates power to be transmitted to the wheel 102, and a brake mechanism 105 that generates a braking force on the wheels 102 and 103. The vehicle 100 also includes a driving operation unit 106 that receives driving operations from the driver, and a vehicle controller 110 that controls the operation of the power source 104 and the brake mechanism 105. The vehicle 100 also includes a positioning device 113 that performs positioning, and a notification device 120 (e.g., a display panel) that notifies the driver of information.

[0012] The power source 104 includes, for example, an internal combustion engine, an electric motor, or both. The driving operation unit 106 includes an accelerator operation unit (such as an accelerator pedal), a brake operation unit (such as a brake pedal), and a steering unit (such as a steering wheel). The brake mechanism 105 includes a mechanism that generates braking force through friction, a device that generates regenerative braking by regeneratively operating an electric motor, or both. The vehicle controller 110 is composed of one or more ECUs (Electronic Control Units). The vehicle controller 110 receives operation signals from the accelerator operation unit, operation signals from the brake operation unit, and driving requests from the controller 20, which will be described later. Then, in response to the operation signals and driving requests, the vehicle controller 110 controls the driving of the power source 104 and the driving of the brake mechanism 105.

[0013] The traffic light information determination device 1 includes a photographing unit 40 that captures an image of the periphery of the vehicle 100, and a controller 20 that determines the display of a target traffic light based on a reflected image of the target traffic light included in the image of a reflecting object captured by the photographing unit 40. The target traffic light refers to a traffic light that displays signal information for the lane in which the vehicle 100 is traveling.

[0014] The photographing unit 40 is a digital camera or the like having a lens and an image sensor. The photographing unit 40 sends a digital video signal to the controller 20. The photographing unit 40 is attached so that the side of the vehicle 100 (the side having width in the front, rear, top and bottom directions) is the photographing direction.

[0015] The controller 20 is composed of one ECU or multiple ECUs that communicate with each other and operate in cooperation with each other. A control program is stored in the controller 20. By executing the control program, the controller 20 extracts the reflected image of the target traffic light from the video of the reflecting object and determines the display of the target traffic light based on the reflected image.

[0016] The traffic light information determination device 1 further includes a front photographing unit 45 that captures an image of the area ahead of the vehicle 100, and the controller 20 may determine the display of the target traffic light from the image if there is no obstruction in the image of the area ahead. The front photographing unit 45 is configured similarly to the photographing unit 40, and is attached so that the photographing direction is the front of the vehicle 100 (the front having width in all directions).

[0017] The traffic light information determination device 1 further includes a first detector 11 capable of detecting the location of a structure located in front of the vehicle 100, and a second detector 12 capable of detecting the location of a structure located to the side of the vehicle 100. The detection information of the first detector 11 and the detection information of the second detector 12 are sent to the controller 20.

[0018] The traffic light information determination device 1 may include an information acquisition unit 15 instead of the first detector 11 and the second detector 12, or in addition to the first detector 11 and the second detector 12. The information acquisition unit 15 acquires information from outside the traffic light information determination device 1 and transfers it to the controller 20. The information acquisition unit 15 corresponds to the first information acquisition unit and the second information acquisition unit according to the present invention.

[0019] The first detector 11 includes, for example, an imaging unit that captures video in the detection direction and a ranging device that can detect the distance to each part in the captured video. A LiDAR (Light Detection and Ranging) or the like can be used as the ranging device. The first detector 11 is attached to the vehicle 100 so that the detection direction is the front of the vehicle 100 (the front having widths in the left, right, top, and bottom directions). From the video information sent from the first detector 11, the controller 20 can calculate the direction of a structure included in the video relative to the vehicle 100 from its position in the video. Furthermore, the controller 20 can obtain the distance to the structure from the distance information sent from the first detector 11. From these angles and distances, the controller 20 can determine the relative position of the structure with respect to the vehicle 100.

[0020] The first detector 11 is not limited to the above configuration. For example, the first detector 11 may be a compound eye camera that can acquire an image and detect the distance to each part in the image using parallax. Alternatively, the first detector 11 may be a distance measuring device that performs 3D scanning. From the profile of distance information obtained by the 3D scan, it is possible to detect where and what kind of structure is located within the scan range, and from the distance information, it is possible to detect the distance to the structure. In addition, various devices can be applied as the first detector 11 as long as they are capable of detecting the location of a structure located in front of the vehicle 100.

[0021] The second detector 12 can have the same configuration as the first detector 11, except for the detection direction. The second detector 12 is installed so that the detection direction is the side of the vehicle 100 (a side having width in the front-rear and top-bottom directions).

[0022] Note that if the first detector 11 is capable of detecting the sides of the vehicle 100 in addition to the front of the vehicle 100, the first detector 11 may be used as the second detector 12. That is, the first detector 11 and the second detector 12 may be combined into one detector. Furthermore, if the second detector 12 is configured to be able to acquire images, the second detector 12 may be used as the imaging unit 40. Similarly, if the first detector 11 is configured to be able to acquire images, the first detector 11 may be used as the front imaging unit 45.

[0023] The information acquisition unit 15 is a communication device capable of communicating with a system having a database 111 and receiving information from the database 111. The database 111 may be provided in the vehicle 100 or in a server device outside the vehicle. Furthermore, the information acquisition unit 15 is a communication device capable of successively receiving positioning information from the positioning device 113 of the vehicle 100. The information acquired by the information acquisition unit 15 includes map information D1 relating to the arrangement of traffic lights, map information D2 relating to the arrangement of buildings around the road, and position information D3 of the vehicle 100.

[0024] The map information D1 related to the location of traffic lights may be map information that directly indicates the position (latitude, longitude, and altitude) of traffic lights, or may be map information that indirectly indicates the position of traffic lights. Map information refers to information that indicates the map positions (latitude, longitude, etc.) of fixed objects such as roads and buildings. Map information that indirectly indicates the position of traffic lights is, for example, map information that indicates the planar position (latitude and longitude) of traffic lights. Traffic light heights are specified, and standard heights are roughly identified. Therefore, the position of traffic lights, including their heights, can be identified from the above map information. Other map information that indirectly indicates the position of traffic lights may be map information that indicates the positions of intersections and crosswalks. The planar position (latitude and longitude) of traffic lights can be roughly identified from the layout of intersections or crosswalks relative to roads, and the position (latitude, longitude, and altitude) of traffic lights can be roughly identified from this information and the standard height of traffic lights.

[0025] The map information D2 relating to the location of buildings may be map information that can directly or indirectly identify the planar position of the side of a building. The buildings may mainly include buildings facing roads. The side of a building may be a side facing a road. The side information may include information indicating whether the material is reflective, such as glass, and position information (height, orientation, etc.) indicating the location of the reflective material portion.

[0026] The vehicle 100 may be equipped with a navigation system 115, and the database 111 having the map information D1, D2 may be the database of the navigation system 115. The information acquisition unit 15 can read out the map information D1, D2 of the area around the road on which the vehicle 100 is traveling from the database 111. Alternatively, the navigation system 115 may supply the map information D1, D2 of the area around the road on which the vehicle 100 is traveling or the map information D1, D2 of the area around the road on which the vehicle 100 is scheduled to travel to the information acquisition unit 15.

[0027] The position information D3 is a measurement value of the position of the vehicle 100 measured by a positioning device 113 provided in the vehicle 100. The positioning device 113 has a receiver that receives signals from positioning satellites, an acceleration sensor, a gyro sensor, etc., and sequentially measures the current position (latitude and longitude) of the vehicle 100 from the detection results thereof. The positioning device 113 may be a component of the navigation system 115.

[0028] <Operation> Next, an example of specific processing by the controller 20 will be described. The controller 20 executes processing to acquire the position of a target traffic light, processing to acquire the position of a reflecting object around the vehicle 100, and processing to estimate the reflection position of the target traffic light. The controller 20 then determines the display of the target traffic light from the image of the estimated reflection position. Furthermore, to increase the accuracy of the above determination, the controller 20 performs processing to verify whether the reflected image included in the image of the reflecting object represents the target traffic light. Furthermore, the controller 20 performs driving assistance processing based on the determination result of the display of the target traffic light. Next, each of these processes will be described in detail.

[0029] <Process to obtain the position of the target traffic light> FIG. 2 is a diagram illustrating the process of acquiring the position of a target traffic light. The controller 20 acquires the positions of the target traffic lights F1 and F2 based on the detection information of the first detector 11, the acquired information of the information acquisition unit 15 (map information D1 and position information D3 of the vehicle 100), or both. The target traffic lights F1 and F2 are traffic lights that display signal information for the lane in which the vehicle 100 is traveling. The signal information includes information such as stop or go by displaying red, yellow, and green lights. The target traffic lights F1 and F2 are often located ahead of the vehicle 100. The positions acquired by the controller 20 correspond to positions relative to the vehicle 100.

[0030] To acquire the positions of the target traffic lights F1 and F2 at the intersection Rx, the controller 20 first recognizes the intersection Rx and each of the intersecting roads r1 and r2 based on the detection information of the first detector 11. This recognition can be achieved, for example, by performing a pattern matching process using image patterns specific to various intersecting roads to determine whether the image patterns are included in the image obtained from the first detector 11. Next, the controller 20 calculates the distance L to the intersection Rx and the widths x and y of each of the intersecting roads r1 and r2 based on the detection information of the first detector 11. Through this calculation, the controller 20 can identify the layout of the intersection Rx and the relative positions of each part of the intersection Rx from the vehicle 100. The controller 20 stores data indicating the standard positions and standard heights of traffic lights at intersections with various layouts. Based on this data, the controller 20 calculates the relative positions of each of the traffic lights F1 to F8 from the vehicle 100 by applying standard positions of traffic lights to the identified layout of the intersection Rx. The controller 20 can then extract information on the relative positions of the target traffic signals F1 and F2 from these.

[0031] This method has the advantage that the positions of the target traffic lights F1 and F2 can be obtained even when the first detector 11 cannot directly detect the target traffic lights F1 and F2 due to an obstacle, for example, when there is a tall vehicle in front of the vehicle 100.

[0032] The method for acquiring the positions of the target traffic signals F1 and F2 is not limited to the above example. For example, the controller 20 may directly identify the target traffic signals F1 and F2 by performing pattern recognition on the video or three-dimensional scan data of the first detector 11. When performing pattern recognition, the controller 20 may narrow the range of pattern recognition based on traffic signal map information acquired from the information acquisition unit 15. Then, once the target traffic signals F1 and F2 have been identified, the controller 20 may obtain the relative positions of the target traffic signals F1 and F2 from the distance and orientation detection information of the first detector 11.

[0033] The controller 20 may also determine the positions of the target traffic lights F1 and F2 from map information D1 regarding the location of traffic lights and position information D3 of the vehicle 100, both of which are acquired from the information acquisition unit 15. In this case, the controller 20 extracts map information D1 regarding the location of the target traffic lights F1 and F2 located on the route of the vehicle 100 from the map information in the database 111, based on the position information D3 of the vehicle 100. The controller 20 then determines the positions (latitude, longitude, and altitude) of the target traffic lights F1 and F2 on the map from the map information D1. Furthermore, the controller 20 can determine the relative positions of the target traffic lights F1 and F2 from the vehicle 100 by calculating the difference between the positions of the target traffic lights F1 and F2 on the map and the current position of the vehicle 100.

[0034] The controller 20 can also obtain the position of the target traffic light by performing the same process as above at intersections other than the intersection Rx, such as at roads having a crosswalk and traffic lights.

[0035] As the vehicle 100 travels, the relative positions of the target traffic signals F1 and F2 change in real time. The controller 20 can obtain the positions (relative positions) of the target traffic signals F1 and F2 in real time by repeating any of the position acquisition processes described above. Alternatively, once the positions of the target traffic signals F1 and F2 have been obtained using any of the methods described above, the controller 20 may then obtain the real-time positions of the target traffic signals F1 and F2 from changes in the position of the vehicle 100. That is, the controller 20 represents the obtained positions of the target traffic signals F1 and F2 as position vectors. The controller 20 can then obtain the real-time positions of the target traffic signals F1 and F2 by sequentially subtracting the displacement vector (amount and direction of movement) of the vehicle 100 from the position vector. The displacement vector of the vehicle 100 can be calculated from the positioning information of the positioning device 113.

[0036] <Process to obtain the position of a reflecting object> 3 is a diagram illustrating the process of acquiring the position of a reflecting object. The controller 20 acquires the position of a reflecting object located around the vehicle 100 based on the detection information of the second detector 12, the acquired information of the information acquisition unit 15 (map information D2 and position information D3 of the vehicle 100), or both of these. The reflecting object located around the vehicle 100 includes windows w1 to w4 of a building H around the road, another vehicle 200 located around the vehicle 100, etc. The position acquired by the controller 20 corresponds to the relative position from the vehicle 100.

[0037] When acquiring the position of a reflecting object, first, the controller 20 recognizes a building H facing the road and surrounding vehicles 200 by performing pattern recognition on the image or three-dimensional scan data obtained from the second detector 12. Next, the controller 20 extracts the distance and direction to the recognized building H and the distance and direction to the recognized surrounding vehicles 200 from the detection information of the second detector 12, thereby acquiring the relative positions to the building H and the relative positions to the surrounding vehicles 200. Then, assuming that there are reflective surfaces such as windows w1 to w4 on the side of the building H facing the road, the controller 20 acquires the positions of the side surfaces as the positions of the reflecting objects. Furthermore, the controller 20 acquires the positions of the outer surfaces of the surrounding vehicles 200 as the positions of the reflecting objects.

[0038] During pattern recognition, the controller 20 may determine whether or not there are reflective objects such as windows w1 to w4 on the side of the building H, and acquire the positions of only those areas recognized as reflective objects as the positions of the reflective objects. Furthermore, during pattern recognition, the controller 20 may recognize the windows, bodies, and body colors of the surrounding vehicles 200, and acquire the positions of the reflective objects by regarding only the windows and bodies that can reflect the color of traffic lights as reflective objects.

[0039] It should be noted that the method for acquiring the position of the reflecting object is not limited to the above example. For example, the controller 20 may acquire the position of the side of the building H facing the road as the position of the reflecting object from the position information D3 of the vehicle 100 obtained from the information acquisition unit 15 and the map information D2 related to the layout of the buildings. The controller 20 can acquire the relative position from the vehicle 100 to the side of the building by calculating the difference between the information (latitude, longitude, orientation) on the position of the side of the building H obtained from the map information D2 and the position information D3 (latitude, longitude, orientation) of the vehicle 100.

[0040] As the vehicle 100 travels, the relative position of the structure H changes in real time. Furthermore, as time passes, the relative position of the vehicle 100 to the surrounding vehicles 200 also changes in real time. The controller 20 can obtain the position (relative position) of the reflecting object in real time by repeating the above-described position acquisition process. Alternatively, once the position of the side of the structure H is obtained, the controller 20 may then obtain the real-time position of the side of the structure H from changes in the position of the vehicle 100. That is, the controller 20 represents the obtained position of the structure H as a position vector. Then, the controller 20 can obtain the real-time position of the side of the structure H by sequentially subtracting the displacement vector (amount and direction of movement) of the vehicle 100 from the position vector. The displacement vector of the vehicle 100 can be calculated from the positioning information of the positioning device 113.

[0041] <Processing to estimate the reflection position of the target traffic light> 4 is a diagram illustrating the process of estimating the reflection position of the target traffic light. The controller 20 estimates the reflection position c1 of the target traffic lights F1 and F2 based on the acquired positions of the target traffic lights F1 and F2 and the position of the reflecting object.

[0042] For example, the controller 20 calculates, as a candidate for the reflection position c1, a position from the midpoint p1 of the line segment q1 connecting the target traffic signal F1 and the vehicle 100 in a direction perpendicular to the line segment q1, at a height half the sum of the height of the target traffic signal F1 and the height of the imaging unit 40 of the vehicle 100. Then, if a reflecting object is found among the candidates from among the reflecting objects whose positions have been acquired, the controller 20 estimates the position as the reflection position c1 of the target traffic signal F1. This estimation method makes it easy to determine the accurate reflection position c1 of the target traffic signal F1 when the vehicle 100 and the target traffic signal F1 are located on a straight line along the road and the reflecting object has a reflective surface parallel to the road. That is, at the candidate for the reflection position c1, the light of the target traffic signal F1 is reflected so that the angle of incidence α = the angle of reflection β, and the reflected light reaches the vehicle 100. When using this method, the controller 20 may exclude the sides of buildings that are not parallel to the road from the reflecting objects, and may use only the sides of buildings that are parallel to the road as the reflecting objects.

[0043] Alternatively, the controller 20 may estimate the reflection position c1 of the target traffic light F1 as follows. First, the controller 20 sets multiple candidate points for all surrounding reflecting objects acquired by the controller 20. The multiple candidate points should be set so as to be uniformly distributed on the reflecting objects. The multiple candidate points may be set to a number large enough to avoid excessive computational load. Next, the controller 20 calculates line segments connecting the image capture unit 40 of the vehicle 100 to each candidate point, as well as line segments of reflected light that reflect through the candidate points when the line segments are regarded as light rays. The reflection direction can be calculated based on the orientation of the reflecting object acquired by the controller 20 so that the reflection angle β = the incident angle α. Furthermore, the controller 20 determines whether the line segments of the reflected light overlap the target traffic light F1. If they do, the controller 20 estimates the candidate point as the reflection position c1 of the target traffic light F1. By performing such calculations for all candidate points, the controller 20 can estimate one or more reflection positions c1 for the target traffic light F1.

[0044] When there are multiple target traffic lights F1, F2, the controller 20 executes the above-described process of estimating the reflection position c1 for each of the multiple target traffic lights F1, F2.

[0045] <Signal display judgment processing and validity confirmation processing> 5 is a diagram illustrating the signal display determination process and validity confirmation process. The following describes a case where a display determination is made for one of the two target signals F1, but the controller 20 can also make a display determination for the other target signal F2 in the same way.

[0046] The image capturing unit 40 captures an image of the side of the vehicle 100 and sends the image data to the controller 20. This image includes an image v1 of the reflection position c1 estimated by the estimation process described above. If, as expected, there is a reflecting object at the reflection position c1 and the acquired position of the target traffic light F1 is normal, the image v1 of the reflection position c1 will include a reflected image Fa of the target traffic light F1. The controller 20 performs traffic light pattern recognition on the image v1 of the reflection position c1. If the result matches the traffic light pattern, the controller 20 determines the display of the target traffic light F1 from the display state (such as the lighting color or lighting position) of the matching reflection image Fa.

[0047] Even if the image ahead is obstructed, for example, because a tall vehicle 220 is located in front of the vehicle 100 or the sun K is overlapping behind the target traffic light F1, the controller 20 can determine the display of the target traffic light F1 by using the reflected image Fa.

[0048] If the pattern recognition process recognizes that the reflected image is a traffic light, the controller 20 then executes a validation process to determine whether the reflected image Fa represents the target traffic light F1. The validation process includes a process of predicting the distorted shape of the target traffic light F1 caused by reflection from a reflecting object, and a process of comparing the predicted distorted shape with the reflected image Fa of the traffic light included in the image v1 at the reflection position c1. The controller 20 verifies the shape of the reflected image Fa through the comparison process.

[0049] Next, a specific example of the validation process will be described.

[0050] The controller 20 stores data on the standard aspect ratio of traffic lights or the standard shape of traffic lights in advance. Alternatively, the controller 20 may extract data on the shape and orientation of the target traffic light F1 from the detection information of the first detector 11 and store the data in advance. Even when an obstruction occurs in the image ahead of the vehicle 100, there is often no obstruction until a short time before the obstruction occurs, and the controller 20 can extract data on the shape and orientation of the target traffic light F1 from the detection information of the first detector 11 during the period when there is no obstruction.

[0051] The reflected image Fa of the traffic signal is an image of the traffic signal viewed from an oblique direction through a reflecting object. Therefore, the aspect ratio of the reflected image Fa of the traffic signal changes from the aspect ratio of the actual traffic signal. The controller 20 predicts, for example, the change in aspect ratio as a process for predicting the distorted shape of the target traffic signal F1. The controller 20 calculates the change in aspect ratio as a distorted shape based on the actual shape and orientation of the target traffic signal F1 and the relative positions of the vehicle 100, the reflecting object, and the target traffic signal F1. The controller 20 then compares the aspect ratio of the traffic signal pattern-recognized in the reflected image Fa with the calculated distorted shape and determines whether the difference between them is within a predetermined allowable error. If the difference is within the allowable error, the controller 20 determines that the validity of the judgment result of the display of the target traffic signal F1 is high. On the other hand, if the difference exceeds the allowable error, the controller 20 determines that the validity of the judgment result of the display of the target traffic signal F1 is low.

[0052] The validity confirmation process is not limited to the process based on comparing the shape of the reflected image Fa described above, and the following validity confirmation process may be performed instead of or in addition to the process. For example, the controller 20 detects the driving conditions (whether or not a vehicle is driving, its direction of travel, etc.) of surrounding vehicles and compares the driving conditions with the determination result of the display of the target traffic light F1 to see if they match. In this case, if all the conditions match, the controller 20 determines that the validity of the determination result of the display of the target traffic light F1 is high. On the other hand, the more points that do not match, the less validity the controller 20 determines that the validity of the determination result of the display of the target traffic light F1 is.

[0053] Alternatively, as a validity confirmation process, the controller 20 may check the shape of the traffic light's reflected image Fa not only for the target traffic lights F1 and F2, but also for the other traffic lights F3 to F8 located in the vicinity. The controller 20 may then compare the shape of the traffic light in the reflected image Fa with the distorted shape predicted when the other traffic lights F3 to F8 are reflected, and determine whether the two are close to each other (for example, whether the aspect ratio is within a predetermined error). In this case, if they are close, the controller 20 determines that the validity of the judgment result for the display of the target traffic light F1 is low. On the other hand, if they are not close, the controller 20 determines that the validity of the judgment result for the display of the target traffic light F1 is high.

[0054] By performing the above-described validity confirmation process, even if, for example, the image v1 of the reflecting object contains a reflected image of another traffic light F7, the controller 20 can reduce the chance of incorrectly determining the display of the target traffic light F1 from the reflected image.

[0055] <Driving assistance processing> The controller 20 may perform driving assistance processing based on the determination result of the display of the target traffic light F1. When performing driving assistance processing, the controller 20 may be provided with detection information from various detectors that detect the driving conditions of the vehicle 100 (vehicle speed, steering angle, accelerator operation amount, brake operation amount, etc.). In addition, the controller 20 may be provided with monitoring result information from a device that monitors the situation around the vehicle 100 (presence or absence of people, vehicles, other objects, etc.). Furthermore, the controller 20 may be capable of controlling a notification device 120 (e.g., a display panel) that notifies the driver of information. Then, based on the determination result of the display of the target traffic light F1 and the above-mentioned detection information and monitoring result information, the controller 20 sends a driving request corresponding to these to the vehicle controller 110 as driving assistance. Furthermore, the controller 20 outputs a notification corresponding to the driving situation from the notification device 120 as driving assistance. Specifically, when the target traffic light F1 indicates a stop and the vehicle 100 is traveling, the controller 20 may send a driving request to the vehicle controller 110 to decelerate the vehicle 100. Furthermore, the controller 20 may output from the notification device 120 the driving request sent to the vehicle controller 110, the determination result of the display of the target traffic light F1, and the like.

[0056] <Control flow> Next, a series of control flows by the traffic light information determination device 1 will be described. Fig. 6 is a flowchart showing an example of control processing by the traffic light information determination device. The control processing is started, for example, when the driver turns on a driving assistance function. When the control processing is started, the controller 20 operates the first detector 11 and the second detector 12 to obtain detection information in front of and to the sides of the vehicle 100 (step S1).

[0057] Furthermore, the controller 20 determines whether there is an intersection near the front of the vehicle 100 (step S2). Specifically, the controller 20 determines whether the distance to the upcoming intersection is equal to or less than a threshold. The determination in step S2 may be made by the controller 20 analyzing the detection information of the first detector 11, or by the controller 20 analyzing the position information D3 of the vehicle 100 and map information of the road. Alternatively, the determination in step S2 may be made by the controller 20 based on a notification from a navigation system of the vehicle 100 (notification that the vehicle is approaching an intersection). Note that the determination process in step S2 is not limited to an intersection, and may be a process for determining proximity to a road location having a traffic light.

[0058] If the determined result in step S2 is NO, the controller 20 returns the process to step S1 and repeats the processes of steps S1 and S2. If the determined result in step S2 is YES, the controller 20 proceeds with the process.

[0059] As a result, as the process progresses, the controller 20 performs a display determination of the target traffic light from the image acquired by the forward photographing unit 45 (step S3). Next, the controller 20 determines whether the result of the determination in step S3 is normal (OK) or bad (NG) (step S4). If a problem occurs in the forward image, the determination result is determined to be bad (NG).

[0060] If the determination result in step S4 is OK, the controller 20 performs one cycle of driving assistance processing using the display determination result of the target traffic light (step S12), and returns to step S1. The driving assistance processing in step S12 corresponds to a portion of the series of driving assistance processing. For example, if driving assistance is performed throughout a series of periods from when the vehicle 100 approaches an intersection until it passes the intersection, the processing in step S5 corresponds to one cycle of processing into which the series of periods is divided. Therefore, if there is no obstruction to the forward image from when the vehicle 100 approaches the intersection until it passes the intersection, the loop processing of steps S1 to S4 and S12 continues, and the driving assistance processing of step S12 is repeatedly executed within the loop processing. This repeated execution realizes a series of driving assistance according to the display of the target traffic light. The driving assistance processing is as described above.

[0061] On the other hand, if the result of the determination in step S4 is NG, the controller 20 executes a process to acquire the position of the target traffic light (step S5), a process to acquire the position of a surrounding reflecting object (step S6), and a process to estimate the reflection position of the target traffic light (step S7). Next, the controller 20 executes a process to determine the display of the target traffic light from the image of the reflection position (step S8), and determines whether the display determination was normal (OK) or defective (NG) (step S9). If the result is OK, the controller 20 executes a process to check the validity of the determination result (step S10), and determines whether the validity is equal to or greater than a threshold (step S11). The processes in steps S5 to S8 and S10 are as described above.

[0062] In the process of acquiring the position of the target traffic light in step S5, an obstruction in the forward image may make it difficult to recognize the target traffic light from the detection information of the first detector 11. In this case, the controller 20 may calculate the position of the target traffic light using the detection information of the first detector 11 received in step S1 before the obstruction occurs in the forward image and the displacement vector of the vehicle 100 thereafter.

[0063] If the determination result of step S9 is OK and the determination result of step S11 is YES, the controller 20 executes one cycle of driving assistance processing using the result of the display determination of the target traffic light (step S12). Then, the controller 20 returns the processing to step S1. Therefore, even if an obstruction occurs in the image ahead, as long as a reflected image of the target traffic light is obtained from a surrounding reflector, the loop processing of steps S1 to S12 continues, and the driving assistance processing of step S12 is repeatedly executed within the loop processing. Then, by this repetition, a series of driving assistance according to the display of the target traffic light is realized.

[0064] On the other hand, if the determination result of step S9 is NG or the determination result of step S11 is NO, the controller 20 stops the driving assistance function and performs a process of notifying the driver of the stoppage (step S13). Then, the control process ends. The driver can resume the control process by turning on the driving assistance function again. Alternatively, after the process of step S12, if a predetermined condition is satisfied, the controller 20 may return the process to step S1.

[0065] The control program of Fig. 6 is stored in a non-transitory computer readable medium such as a ROM included in the controller 20. The controller 20 may be configured to read and execute a program stored in a portable non-transitory recording medium. The portable non-transitory recording medium may store the control program described above.

[0066] As described above, the traffic light information determination device 1 of this embodiment includes the photographing unit 40 that acquires video and the controller 20 that determines the display of the target traffic light. The photographing unit 40 acquires video of a reflecting object located around the vehicle 100. The controller 20 then determines the display of the target traffic light based on the reflected image of the target traffic light contained in the video of the reflecting object. Therefore, even if the target traffic light is located ahead of the vehicle 100 and obstructs the video ahead of the vehicle 100, the controller 20 can determine the display of the target traffic light.

[0067] Furthermore, according to the traffic light information determination device 1 of this embodiment, the controller 20 estimates the reflection position of the target traffic light based on the position of the target traffic light and the position of the reflecting object, and determines the display of the target traffic light based on the image of the estimated reflection position. Therefore, the controller 20 can omit processing to search for the reflected image of the target traffic light to an extent where the reflected image of the target traffic light is not located, thereby reducing the load on the controller 20 and improving the speed and accuracy of recognizing the reflected image of the target traffic light.

[0068] In the above embodiment, an example has been described in which a process for estimating the reflection position of a target traffic light is executed. However, the controller 20 may be configured to find the reflected image of the target traffic light without performing a process for estimating the reflection position. For example, the controller 20 can find the reflected image of the target traffic light without estimating the reflection position by performing pattern recognition processing of the reflected image of the traffic light over a wide area of ​​the image captured by the image capture unit 40. Alternatively, the controller 20 can find the reflected image of the target traffic light by acquiring the positions of reflecting objects around the vehicle 100 and performing pattern recognition processing of the reflected image of the traffic light over the entire area in which the reflecting objects are located. For traffic lights on a road without branching off, the traffic light facing the display side corresponds to the target traffic light. Therefore, even without estimating the reflection position, if the controller searches for the reflected image of a traffic light facing the display side using pattern recognition, the traffic light corresponds to the target traffic light, making it possible to distinguish the target traffic light from a traffic light in another lane.

[0069] Furthermore, the traffic light information determination device 1 of this embodiment includes a first detector 11 capable of detecting the location of structures ahead and a second detector 12 capable of detecting the location of structures to the side. The traffic light information determination device 1 also includes an information acquisition unit 15 capable of acquiring map information D1 relating to the location of traffic lights, map information D2 relating to the location of buildings, and position information D3 of the vehicle 100. The controller 20 acquires the position of the target traffic light based on the detection information of the first detector 11, the map information D1 and position information D3 acquired by the information acquisition unit 15, or both. The controller 20 also acquires the position of the reflecting object based on the detection information of the second detector 12, the map information D2 and position information D3 acquired by the information acquisition unit 15, or both. This improves the accuracy of the process for acquiring the position of the target traffic light and the process for acquiring the position of the reflecting object.

[0070] Furthermore, according to the traffic light information determination device 1 of this embodiment, the controller 20 further executes a validation process to determine whether the image recognized as the reflected image of the target traffic light actually represents the target traffic light. By referring to the result of the validation process, if the determination result of the display of the target traffic light is incorrect, the controller 20 can reduce the possibility of using the incorrect determination result for other control.

[0071] Furthermore, according to the traffic light information determination device 1 of this embodiment, the validity confirmation process includes a process of predicting the distorted shape of the target traffic light due to reflection, and a process of comparing the image recognized as the reflected image of the target traffic light with the predicted distorted shape. This validity confirmation process allows the controller 20 to efficiently reduce the chance of confusing the target traffic light in the lane of the vehicle 100 at an intersection or the like with a traffic light on another intersecting road.

[0072] The above describes the embodiments of the present invention. However, the present invention is not limited to the above embodiments. For example, in the above embodiments, an example was shown in which the traffic light information determination device 1 performs driving assistance processing. However, the traffic light information determination device 1 may not perform driving assistance processing, but may instead supply the signal display determination result to another driving assistance device. When performing driving assistance processing, the traffic light information determination device 1 may also be called a driving assistance device. Other details shown in the embodiments may be modified as appropriate without departing from the spirit of the invention. [Explanation of symbols]

[0073] 1 Traffic light information determination device 11 First detector 12 Second detector 15 Information acquisition department 20 Controller 40 Photography Department 45 Frontal shooting section 100 vehicles 102, 103 wheels 104 Power source 105 Brake mechanism 106 Driving operation unit 110 Vehicle Controller 111 Database 113 Positioning Device 115 Navigation System 120 Notification device F1, F2 target traffic lights H Building c1 Reflection position Fa Reflection image

Claims

1. A traffic light information determination device mounted on a vehicle, a photographing unit that acquires video; a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and Equipped with The photographing unit acquires an image of a front of the vehicle and an image of a reflecting object located around the vehicle, The traffic light information determination device is characterized in that, when a normal determination result of the display of the target traffic light cannot be obtained based on the image of the front, the controller determines the display of the target traffic light based on the reflected image of the target traffic light contained in the image of the reflecting object.

2. A traffic light information determination device mounted on a vehicle, a photographing unit that acquires video; a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and Equipped with The photographing unit acquires an image of a reflecting object located around the vehicle, The controller estimates the reflection position of the target traffic light based on the position of the target traffic light and the position of the reflecting object, and determines the display of the target traffic light based on the reflected image of the target traffic light included in the image of the reflection position of the reflecting object.

3. A traffic light information determination device mounted on a vehicle, a photographing unit that acquires video; a controller that determines the display of a target traffic light that displays signal information for the lane in which the vehicle is traveling, and Equipped with The photographing unit acquires an image of a reflecting object located around the vehicle, The controller determining the display of the target traffic signal based on a reflected image of the target traffic signal included in the image of the reflecting object; Furthermore, a validation process is performed to determine whether the reflected image represents the target traffic light; The validation process includes: A process of predicting a distorted shape of the target traffic light due to reflection; comparing the reflected image with the predicted distorted shape; A traffic light information determination device comprising:

4. a first detector capable of detecting the location of a structure located in front of the vehicle, a first information acquisition unit capable of acquiring map information relating to the location of traffic signals and location information of the vehicle, or both of these; a second detector capable of detecting the location of a structure located on the side of the vehicle, a second information acquisition unit capable of acquiring map information relating to the location of the structure and location information of the vehicle, or both of these; Equipped with A traffic light information determination device as described in any one of claims 1 to 3, characterized in that the controller acquires the position of the target traffic light based on the detection information of the first detector, the acquired information of the first information acquisition unit, or both, and acquires the position of the reflecting object based on the detection information of the second detector, the acquired information of the second information acquisition unit, or both.

Images