Driving assistance device, driving assistance method, and program

Abstract

To provide a device and a method for supporting drive and a program that can precisely support drive according to an instruction to go forward by a traffic signal.SOLUTION: A drive support device includes: a recognition unit for recognizing at least presence and a state of a traffic signal on the basis of an image of a region around a vehicle; a drive support processing unit for supporting drive for a passenger of the vehicle on the basis of at least, the state of the traffic signal. The drive support processing unit controls timing of supporting drive according to a lighting color of the traffic signal in the recognized state.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a driving assistance device, a driving assistance method, and a program. [Background technology]

[0002] There is a technology for controlling vehicle speed and providing driving assistance such as alerting vehicle occupants based on a traffic light's progress indication obtained by recognizing an image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-36100 A Summary of the Invention [Problem to be solved by the invention]

[0004] By providing driving assistance based on highly reliable proceeding suggestions obtained by image recognition, it is possible to provide assistance that is useful to occupants with greater accuracy. However, the recognition rate of traffic lights using image recognition varies depending on the lighting color. Therefore, there are cases where the reliability of proceeding suggestions represented by a specific lighting color is low.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide a driving assistance device, a driving assistance method, and a program that can provide driving assistance in response to traffic light advancement suggestions with greater accuracy. [Means for solving the problem]

[0006] A driving assistance device, a driving assistance method, and a program according to the present invention employ the following configuration. (1): A driving assistance device according to one embodiment of the present invention includes a recognition unit that recognizes the presence and state of at least a traffic light based on an image of the vehicle's surroundings, and a driving assistance processing unit that provides driving assistance to an occupant of the vehicle based on at least the state of the traffic light, and the driving assistance processing unit controls the timing of providing the driving assistance in accordance with the illumination color of the traffic light, which is included in the state.

[0007] (2): In the above aspect (1), the driving assistance processing unit is provided with an alarm control unit that instructs an alarm device that alerts the occupant to output information about the traffic light based on the lighting color.

[0008] (3) In the aspect (1) above, the driving assistance processing unit includes a driving control unit that controls acceleration / deceleration or steering of the vehicle in response to the lighting color.

[0009] (4): In the above aspect (1), the recognition unit has an area identification unit that identifies a traffic light recognition area, which is the area of ​​the traffic light in the image, and a proceeding suggestion estimation unit that estimates a proceeding suggestion of the traffic light using at least color information in the traffic light recognition area and outputs it as at least a part of the state of the traffic light.

[0010] (5): In any of the above aspects (1) to (4), the recognition unit has an area identification unit that identifies a traffic light recognition area, which is the area of ​​a traffic light in the image, and when a yellow light is detected in the traffic light recognition area of ​​the image, the recognition unit stores the traffic light recognition area as a traffic light area candidate, and when the traffic light area candidate matches a predetermined amount or more in multiple chronologically consecutive images, the recognition unit increases the certainty of the recognition of the traffic light in the traffic light area candidate.

[0011] (6): In any of the above aspects (1) to (4), the recognition unit has an area identification unit that identifies a traffic light recognition area, which is the area of ​​a traffic light in the image, and detects the lighting color of a traffic light in the traffic light recognition area of ​​the image. The driving assistance processing unit performs driving assistance based on the yellow lighting color when a yellow lighting color is detected in the traffic light recognition area of ​​the image and a green lighting color traffic light is detected in an area corresponding to the traffic light recognition area in another image obtained prior to the image.

[0012] (7): In any of the aspects (1) to (4) above, the driving assistance processing unit performs the driving assistance when the distance between the vehicle and the traffic light is within a distance corresponding to the speed of the vehicle and the traffic light is red, and does not perform the driving assistance when the distance between the vehicle and the traffic light is within a distance corresponding to the speed of the vehicle and the traffic light is yellow.

[0013] (8): A driving assistance method according to another aspect of the present invention is a driving assistance method in which a computer recognizes the presence and state of at least traffic lights based on an image of the surroundings of the vehicle, provides driving assistance to an occupant of the vehicle based on at least the state of the traffic lights, and controls the timing of providing the driving assistance according to the illumination color of the traffic lights included in the state.

[0014] (9): A program according to another aspect of the present invention is a program that causes a computer to recognize at least the presence and state of traffic lights based on an image of the vehicle's surroundings, provides driving assistance to an occupant of the vehicle based on at least the state of the traffic lights, and controls the timing of providing the driving assistance depending on the illumination color of the traffic lights included in the state. Effect of the Invention

[0015] According to (1) to (9), driving assistance based on traffic light information detected from images can be provided at a timing according to the likelihood of a progression indication indicated by the traffic light color obtained by image recognition.

[0016] According to (2), traffic light information according to the color of the traffic light can be notified at a timing according to the likelihood of a proceeding indication.

[0017] According to (3), the vehicle acceleration / deceleration or steering control according to the color of the traffic light can be performed at a timing according to the likelihood of the traffic light indicating that the vehicle is moving forward.

[0018] According to (4), it is possible to accurately estimate information indicating a direction in which a vehicle can proceed based on the color of a traffic signal. Also, it is possible to accurately estimate information indicating a direction in which a vehicle can proceed based on the color of a traffic signal.

[0019] According to (5) and (6), the accuracy of recognition of a traffic light can be increased even for a traffic light that is lit in yellow, which has a lower accuracy in image recognition than other colors.

[0020] According to (7), when the light color is recognized as yellow, the driving assistance that is executed immediately when the traffic light color is detected as red can be avoided, and the driving assistance can be executed after the light color changes to red, which has a higher recognition rate than when the traffic light color is yellow. [Brief description of the drawings]

[0021] [Figure 1] 1 is a configuration diagram of a vehicle system according to an embodiment of the present invention. [Diagram 2] 3 is a diagram showing the lighting colors of traffic signals detected by the driving assistance device according to the embodiment. FIG. [Diagram 3] 3 is a diagram showing the lighting colors of traffic signals detected by the driving assistance device according to the embodiment. FIG. [Figure 4] 2 is a diagram showing a traffic light recognition area detected by the driving assistance device according to the embodiment. FIG. [Diagram 5] 4 is a flowchart showing a process of the driving assistance device according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, embodiments of a driving assistance device, a driving assistance method, and a program according to the present invention will be described with reference to the drawings.

[0023] 1 is a configuration diagram of a vehicle system 1 that utilizes a driving assistance device according to an embodiment. The vehicle on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0024] The vehicle system 1 includes, for example, an external sensor 10, an object recognition device 16, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a driving operator 80, a driving support device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other by multiple communication lines such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configuration may be added.

[0025] The external sensor 10 acquires information on the external environment of the vehicle (hereinafter, the vehicle M) on which the vehicle system 1 is mounted. The external sensor 10 includes a camera 11, a radar device 12, and a LIDAR (Light Detection and Ranging) 14. The camera 11 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 11 is attached to any position of the vehicle M. When capturing an image of the front, the camera 11 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 11, for example, periodically and repeatedly captures images of the surroundings of the vehicle M. The camera 11 may be a stereo camera.

[0026] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to an arbitrary location of the vehicle M. The radar device 12 may detect the position and speed of an object by an FM-CW (Frequency Modulated Continuous Wave) method.

[0027] The LIDAR 14 irradiates light (or electromagnetic waves with a wavelength close to that of light) around the vehicle M and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, a pulsed laser light. The LIDAR 14 is attached to an arbitrary location of the vehicle M.

[0028] The object recognition device 16 performs sensor fusion processing on some or all of the detection results from the camera 11, the radar device 12, and the LIDAR 14 to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition result to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 11, the radar device 12, and the LIDAR 14 to the driving assistance device 100 as they are.

[0029] The HMI 30 is an information device that notifies various information to an occupant of the vehicle M. The HMI 30 also accepts input operations by the occupant. The HMI 30 includes various display devices, a speaker, a buzzer, a touch panel, switches, keys, and the like.

[0030] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the host vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, a direction sensor that detects the direction of the host vehicle M, and the like.

[0031] The driving controls 80 include, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a special steering wheel, a joystick, and other controls.

[0032] The driving support device 100 includes a recognition unit 120, a driving support processing unit 140, and a storage unit 160. The recognition unit 120 and the driving support processing unit 140 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (storage device having a non-transient storage medium) such as an HDD or flash memory of the driving support device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving support device 100 by mounting the storage medium (non-transient storage medium) in a drive device.

[0033] The recognition unit 120 performs image recognition on the image of the surroundings of the vehicle M captured by the camera 11, and recognizes the object. The recognition unit 120 obtains the type of object captured in the image and a score indicating the likelihood that the object is of that type through image recognition. Any conventional technology can be used for image recognition. For example, the recognition unit 120 may realize this image recognition by performing a function based on AI (Artificial Intelligence) and a function based on a pre-given model in parallel. For example, when recognizing a traffic light, the recognition unit 120 performs recognition of the traffic light by deep learning or the like in parallel with recognition of the traffic light by pattern matching, and obtains an overall score based on the scores of both recognitions. Also, for example, when recognizing an intersection, the recognition unit 120 performs recognition of the intersection by deep learning or the like in parallel with recognition based on pre-given conditions (such as traffic lights and road markings that can be pattern matched), and obtains an overall score based on the scores of both recognitions. Note that the recognition unit 120 may perform either image recognition using AI or image recognition using a pre-given model.

[0034] Furthermore, the recognition unit 120 recognizes the position, speed, acceleration, and other status of objects around the vehicle M based on information input from the camera 11, the radar device 12, and the LIDAR 14 via the object recognition device 16. The position of the object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for driving assistance. The driving assistance includes notification to the occupants and driving control of the vehicle M. If the recognized object is a traffic light, the status includes information suggesting proceeding.

[0035] Furthermore, the recognition unit 120 recognizes, for example, the lane (driving lane) in which the host vehicle M is traveling. For example, the recognition unit 120 recognizes the driving lane by comparing a pattern of road dividing lines (for example, an arrangement of solid lines and dashed lines) obtained from map information with a pattern of road dividing lines around the host vehicle M recognized from an image captured by the camera 11. The map information is stored in a storage device (not shown) that is provided internally or externally. Note that the recognition unit 120 may recognize the driving lane by recognizing road boundaries (road boundaries) including not only road dividing lines but also road shoulders, curbs, medians, guardrails, and the like. Furthermore, the recognition unit 120 recognizes stop lines, obstacles, red lights, toll booths, and other road phenomena.

[0036] When recognizing the driving lane, the recognition unit 120 recognizes the position and attitude of the host vehicle M with respect to the driving lane. For example, the recognition unit 120 may recognize the deviation of the reference point of the host vehicle M from the center of the lane and the angle with respect to a line connecting the centers of the lanes in the traveling direction of the host vehicle M as the relative position and attitude of the host vehicle M with respect to the driving lane. Alternatively, the recognition unit 120 may recognize the position of the reference point of the host vehicle M with respect to either side end of the driving lane (a road dividing line or a road boundary) as the relative position of the host vehicle M with respect to the driving lane.

[0037] The recognition unit 120 includes an area identification unit 122 and a proceeding indication estimation unit 124. The area identification unit 122 recognizes an area in the image that is estimated to be occupied by a traffic light as a traffic light recognition area. The area identification unit 122 recognizes an area that has the outline of a traffic light and has a light-emitting part of a predetermined color as a traffic light area by a pattern matching or machine learning method. This recognition is not definitive, but is performed together with a certainty (likelihood). The proceeding indication estimation unit 124 estimates a proceeding indication of the traffic light using color information in the traffic light recognition area identified by the area identification unit 122 (determined to have a certainty equal to or greater than a threshold). The proceeding indication is represented by the lighting color of the traffic light. In addition, the proceeding indication of an arrow-type traffic light may be represented by a combination of the color of the arrow and a proceeding direction according to the shape of the arrow.

[0038] The driving support processing unit 140 performs driving support for the vehicle occupants based on the state of the traffic light recognized by the recognition unit 120. The driving support processing unit 140 also controls the timing of providing driving support according to the information on the lighting color included in the state of the traffic light. The score of the recognition result when the lighting color of the traffic light is yellow is lower than the score when the lighting color is other colors. This is because street lights are easily recognized as traffic lights. Therefore, when the recognized lighting color is yellow, the driving support processing unit 140 suppresses driving support based on the suggestion to proceed from the traffic light because the likelihood of the suggestion to proceed is also low. Furthermore, when the recognized lighting color is yellow, the driving support processing unit 140 increases the likelihood of detecting the traffic light by combining the recognition result in the previous image. In this case, the driving support processing unit 140 may not suppress driving support even when the lighting color of the traffic light is yellow.

[0039] The driving support processing unit 140 includes a notification control unit 142 and a driving control unit 144. The notification control unit 142 controls notifications to the occupants of the host vehicle M by the HMI 30. The notification control unit 142 causes information based on the recognition result of the recognition unit 120 to be displayed on the screen of the HMI 30, or to be output as sound from the speaker of the HMI 130. As an example, the notification control unit 142 causes the HMI 30 to output information according to the recognition result of the traffic light's advance suggestion and the speed of the host vehicle M.

[0040] The driving control unit 144 performs driving control of the host vehicle M. Driving control means, for example, controlling one or both of the speed and steering of the host vehicle M. The driving control unit 144 generates a target trajectory along which the host vehicle M will automatically (without the driver's operation) travel in the future so that the host vehicle M can travel in the recommended lane and further respond to the surrounding conditions of the host vehicle M. The surrounding conditions of the host vehicle M are obtained based on the recognition result of the recognition unit 120 and the detection result of the vehicle sensor 40. The surrounding conditions of the host vehicle M include information on the progress indication of the traffic light estimated by the progress indication estimation unit 124. The recommended lane is determined, for example, based on the destination and map information. The destination information may be received, for example, from a navigation device. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a sequence of points (trajectory points) to be reached by the host vehicle M. The trajectory points are points to be reached by the host vehicle M at predetermined travel distances (for example, about several [m]) along the road, and are determined so as to change to the correct lane. Separately from that, a target speed and a target acceleration are generated for each predetermined sampling time (for example, about a few tenths of a second) as part of the target trajectory. Also, the trajectory points may be positions to be reached by the host vehicle M at each sampling time. In this case, the information on the target speed and the target acceleration is expressed by the intervals between the trajectory points.

[0041] The traveling control unit 144 controls the traveling drive force output device 200, the brake device 210, and the steering device 220 so that the host vehicle M passes the target trajectory at the scheduled time. That is, the traveling control unit 144 controls the traveling drive force output device 200 or the brake device 210 based on the speed element associated with the target trajectory. Furthermore, the traveling control unit 144 controls the steering device 220 according to the curvature of the target trajectory. These processes are realized, for example, by a combination of feedforward control and feedback control. As an example, the traveling control unit 144 executes a combination of feedforward control according to the curvature of the road ahead of the host vehicle M and feedback control based on the deviation from the target trajectory.

[0042] As described above, the driving control unit 144 generates a target trajectory using information on the traffic light's proceeding indication estimated by the proceeding indication estimation unit 124, and controls the brake device 210 and the steering device 220 based on the generated target trajectory. The driving control unit 144 also controls the brake device 210 based on the recognition result of the recognition unit 120 and the detection result of the vehicle sensor 40. For example, the driving control unit 144 controls the brake device 210 when the distance between a three-dimensional object and the host vehicle M is within a predetermined distance according to the speed.

[0043] The storage unit 160 stores the traffic light recognition area and the proceeding suggestion obtained for each image during a period going back a predetermined time from the present in association with each other.

[0044] The driving force output device 200 outputs a driving force (torque) for the vehicle to travel to the driving wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, and an ECU (Electronic Control Unit) that controls these. The ECU controls the above components according to information input from the travel control unit 144 or information input from the driving operation device 80.

[0045] The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the cruise control unit 144 or information input from the driving operation element 80, so that a brake torque corresponding to a braking operation is output to each wheel. The brake device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of a brake pedal included in the driving operation element 80 to the cylinder via a master cylinder. The brake device 210 may be an electronically controlled hydraulic brake device that controls an actuator according to information input from the cruise control unit 144 to transmit hydraulic pressure in the master cylinder to the cylinder.

[0046] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by applying a force to, for example, a rack and pinion mechanism. The steering ECU drives the electric motor according to information input from the driving operator 80 or information input from the driving control unit 144 to change the direction of the steered wheels.

[0047] 2 and 3 are diagrams showing the lighting colors of traffic lights recognized by the driving support device 100. In FIG. 2, a traffic light A is installed at an intersection ahead of the road on which the vehicle M is traveling. At a position P11, the recognition unit 120 of the driving support device 100 recognizes that the lighting color of the traffic light A is yellow based on an image in front of the vehicle M. The notification control unit 142 of the driving support processing unit 140 selects notification information F according to the lighting color of yellow or red and the speed of the vehicle M. The notification information F is information about the traffic light A. Specifically, the notification information F is information that notifies the driver that the traffic light ahead is a red light, that is, that the next intersection is not permitted to proceed. This is because the time that the traffic light is lit in yellow is short, and the red light is lit immediately after the yellow light, so that whether the lighting color is yellow or red is judged to be an indication of not permitting proceeding. In addition, the notification information F may be selected only when the vehicle M is traveling at a speed at which the acceleration when stopping before the intersection is equal to or greater than a predetermined value.

[0048] On the other hand, when the illumination color of traffic light A is yellow, the driving support processing unit 140 determines that the likelihood of the recognition result that the traffic light is a traffic light is not high enough. Therefore, the driving support processing unit 140 instructs the notification control unit 142 to suppress notification of notification information F. In this case, the notification control unit 142 does not notify, for example, the selected information F. Furthermore, the driving support processing unit 140 instructs the traveling control unit 144 to suppress driving support based on the proceeding suggestion of the traffic light. In this case, the traveling control unit 144 suppresses control of the host vehicle M according to the proceeding suggestion of no proceeding determined based on the illumination color of yellow.

[0049] The recognition unit 120 recognizes that the traffic light A is at the position P12 and that the light color of the traffic light A is red. The notification control unit 142 selects the notification information F again. The driving support processing unit 140 determines that the recognition result that the traffic light A is a traffic light is highly likely because the light color of the traffic light A is red. The driving support processing unit 140 instructs the notification control unit 142 to cancel the suppression of the notification of information based on the traffic light's proceeding suggestion. Furthermore, the driving support processing unit 140 instructs the traveling control unit 144 to cancel the suppression of the driving support based on the traffic light's proceeding suggestion. The notification control unit 142 notifies the selected notification information F. Note that the notification control unit 142 may notify the notification information F when the host vehicle M enters an area away from the traffic light A or a predetermined position inside or outside the intersection where the traffic light A is installed by a distance according to the speed of the host vehicle M. The traveling control unit 144 controls the acceleration / deceleration or steering of the host vehicle M according to the proceeding suggestion of proceeding not permitted determined based on the red light color.

[0050] 3, a traffic light A is installed at an intersection ahead of the road on which the host vehicle M is traveling. At position P21, the recognition unit 120 of the driving assistance device 100 recognizes that the illumination color of traffic light A is green, based on an image of the area ahead of the host vehicle M. Here, the notification control unit 142 determines that there is no notification information corresponding to the illumination color of green.

[0051] At the position P22, the recognition unit 120 recognizes that the lighting color of the traffic light A is yellow based on the image in front of the vehicle M. The notification control unit 142 selects the notification information F in the same manner as in the case of FIG. 2. The driving support processing unit 140 judges that the recognition probability of the traffic light A is high because the lighting color of the recognized traffic light A is yellow, but the lighting color of the traffic light A has previously been recognized as blue in the area where the traffic light A is recognized. Therefore, the driving support processing unit 140 does not instruct the notification control unit 142 or the driving control unit 144 to suppress. The notification control unit 142 notifies the notification information F when the vehicle M enters an area away from the traffic light A or a predetermined position inside or outside the intersection by a distance according to the speed of the vehicle M, even before the proceeding indication estimation unit 124 recognizes the red lighting color. In addition, the driving support processing unit 140 controls the acceleration / deceleration or steering of the vehicle M according to the proceeding indication of proceeding not permitted determined based on the yellow lighting color.

[0052] The recognition unit 120 recognizes that at position P23, there is a traffic light A and that the illumination color of the traffic light A is red. After passing position P23, the notification control unit 142 issues notification information F when the host vehicle M enters an area away from the traffic light A or from a predetermined position inside or outside the intersection by a distance according to the speed of the host vehicle M. In addition, the travel control unit 144 controls the acceleration / deceleration or steering of the host vehicle M according to the progress suggestion of progress not permitted determined based on the illumination color of red.

[0053] FIG. 4 is a diagram showing a traffic light recognition area detected by the driving support device 100. The recognition unit 120 of the driving support device 100 recognizes the presence of a traffic light based on the image G1 captured by the camera 11. The area identification unit 122 identifies traffic light recognition areas R1, R2, and R3, which are areas in the image G1 where traffic lights are recognized. The proceeding suggestion estimation unit 124 estimates that the lighting color of each of the traffic light recognition areas R1, R2, and R3 is yellow based on the colors contained in the area. The proceeding suggestion estimation unit 124 associates the shooting time of the image G1, the traffic light recognition areas R1, R2, and R3, and the lighting colors estimated for each of the traffic light recognition areas, and stores them in the storage unit 160. The area associated with the yellow lighting color indicates that it is a traffic light candidate area that is unlikely to be a traffic light. The proceeding indication estimation unit 124 may add information indicating that the traffic light recognition areas R1, R2, and R3 are traffic light candidate areas to the information on the traffic light recognition areas R1, R2, and R3 and write the information to the storage unit 160. The driving support processing unit 140 determines that the recognition result that the traffic light A is a traffic light is not highly likely because the light color of the recognized traffic light A is yellow. Therefore, the driving support processing unit 140 instructs the notification control unit 142 and the driving control unit 144 to suppress driving support for the host vehicle M according to the light color.

[0054] The recognition unit 120 recognizes the presence of a traffic light based on the image G2 captured by the camera 11 after the image G1. The area identification unit 122 estimates a traffic light recognition area R1' in the image G2. The proceeding indication estimation unit 124 estimates that the lighting color is yellow based on the colors contained in the traffic light recognition area R1'. The driving support processing unit 140 reads out information on the traffic light recognition areas R1, R2, and R3 obtained from the image G1 captured one image before the image G2 from the storage unit 160. The driving support processing unit 140 identifies a traffic light recognition area R1 that matches the traffic light recognition area R1' by a predetermined amount or more among the read traffic light recognition areas R1, R2, and R3. Since the lighting color of the identified traffic light recognition area R1 is stored in the storage unit 160 as yellow, the driving support processing unit 140 increases the likelihood that the traffic light recognition area R1' is a traffic light. Specifically, the driving assistance processing unit 140 increases the image recognition score of the traffic light in the traffic light recognition area R1' by a predetermined amount.

[0055] Fig. 5 is a flow diagram showing the driving assistance control process executed by the driving assistance device 100. The driving assistance device 100 performs the driving assistance control process shown in Fig. 5 every time an image of the area ahead of the vehicle M is input. The recognition unit 120 recognizes an object based on the image of the area ahead of the vehicle M. The recognition unit 120 determines whether or not a traffic light has been recognized as the object (step S105). If the recognition unit 120 does not recognize a traffic light, the process of Fig. 5 ends.

[0056] When it is determined that the recognition unit 120 has recognized a traffic light, the area identification unit 122 uses at least color information of the input image as a determination factor to estimate a traffic light recognition area, which is the area of ​​the traffic light recognized by the recognition unit 120 (step S110). The proceeding suggestion estimation unit 124 estimates a proceeding suggestion based on color information contained in the estimated traffic light recognition area (step S115). The proceeding suggestion is expressed by a lighting color or a combination of a lighting color and a proceeding direction. The proceeding suggestion estimation unit 124 writes recognition information that associates image identification information of the image, the shooting time, the traffic light recognition area, and the proceeding suggestion into the storage unit 160 (step S120). The shooting time may be used as the image identification information. Moreover, information on the shooting order may be used instead of one or both of the image identification information and the shooting time.

[0057] The driving support processing unit 140 determines whether the lighting color of the proceeding indication estimated in step S115 is yellow (step S125). If the driving support processing unit 140 determines that the lighting color is yellow, it sets the traffic light recognition area as a traffic light candidate area (step S130). The driving support processing unit 140 adds flag information indicating that the area is a traffic light recognition area candidate to the recognition information written in the storage unit 160 in step S120.

[0058] The driving support processing unit 140 judges whether the lighting color of the traffic light candidate area in the previous image is blue (step S135). Specifically, the driving support processing unit 140 refers to the storage unit 160, and judges YES in step S135 when the recognition information of the image immediately preceding the image currently being processed contains a traffic light recognition area that is common to the traffic light candidate area at a predetermined ratio or more, and the proceeding suggestion associated with the traffic light recognition area is set to have a lighting color of blue. If the driving support processing unit 140 judges YES in step S135, it executes processing of step S165 described later. Furthermore, if the recognition information of the previous image does not contain a traffic light recognition area that is common to the traffic light candidate area at a predetermined ratio or more, the driving support processing unit 140 judges NO in step S135. Furthermore, if the recognition information of the previous image contains a traffic light recognition area that is common to the traffic light candidate area at a predetermined ratio or more, but the proceeding suggestion is not set to have a lighting color of blue, the driving support processing unit 140 judges NO in step S135.

[0059] If the driving support processing unit 140 judges NO in step S135, it judges whether the traffic light candidate area was also a traffic light candidate area in past images (step S140). Specifically, the driving support processing unit 140 reads out the recognition information of each of the images up to a predetermined number of images before the image to be processed or up to a predetermined time before the image to be processed from the storage unit 160. The driving support processing unit 140 identifies recognition information in which a traffic light recognition area that is common to the traffic light candidate area by a predetermined ratio or more is set and flag information indicating that the area is a traffic light candidate area is set. If the ratio of the identified recognition information among the read recognition information is equal to or greater than a threshold, the driving support processing unit 140 judges YES in step S140, and if the ratio is less than the threshold, it judges NO in step S140.

[0060] If the driving assistance processing unit 140 judges NO in step S140, it outputs a driving assistance suppression instruction to the notification control unit 142 and the driving control unit 144 (step S145). The driving assistance suppression instruction is an instruction to suppress driving assistance based on a progress suggestion. Note that, if the notification control unit 142 and the driving control unit 144 have already suppressed driving assistance, the driving assistance processing unit 140 does not need to transmit the driving assistance suppression instruction.

[0061] If the driving support processing unit 140 judges YES in step S140, it increases the image recognition score of the traffic light detected by the recognition unit 120 by a predetermined increment or a predetermined increment rate (step S150). The increment or rate may be constant, or may increase as the rate calculated in step S140 increases. If the driving support processing unit 140 judges that the increased score is equal to or greater than a threshold, it performs the process of step S165 described below, and if it judges that the increased score is less than the threshold, it performs the process of step S145.

[0062] When the driving support processing unit 140 determines in step S125 that the lighting color of the proceeding suggestion is not yellow, it determines whether the lighting color is red (step S160). When the driving support processing unit 140 determines in step S160 that the lighting color of the proceeding suggestion is red, or when the driving support processing unit 140 determines YES in step S135, it outputs a driving support suppression release instruction to the notification control unit 142 and the driving control unit 144 (step S165). When the notification control unit 142 and the driving control unit 144 receive the driving support suppression release instruction, they release the suppression of driving support based on the lighting color information indicated by the proceeding suggestion. Note that the driving support processing unit 140 does not need to transmit the driving support suppression release instruction when the notification control unit 142 and the driving control unit 144 are not suppressing driving support. The notification control unit 142 and the driving control unit 144 perform driving support according to the proceeding suggestion (step S170). For example, the notification control unit 142 causes the HMI 30 to output a voice message indicating that proceeding to the upcoming intersection is prohibited. Alternatively, the driving control unit 144 controls the braking device 210 and the steering device 220 to stop the host vehicle M in the lane in which the host vehicle M is currently traveling or by changing lanes.

[0063] When the driving support processing unit 140 determines in step S160 that the illumination color of the proceeding suggestion is not red, it outputs a driving support suppression cancellation instruction to the notification control unit 142 and the driving control unit 144 (step S175). Note that the driving support processing unit 140 does not need to transmit a driving support suppression cancellation instruction when the notification control unit 142 and the driving control unit 144 are not suppressing driving support. Also, when step S175 is executed, the illumination color of the proceeding suggestion is blue. Therefore, when there is driving support corresponding to the proceeding suggestion with a blue illumination color, the notification control unit 142 and the driving control unit 144 may perform the driving support. The driving support device 100 ends the processing of FIG. 5.

[0064] According to the embodiment described above, the driving assistance device 100 can change the timing of notifying the driver of the information on the traffic light corresponding to the lighting color and the timing of starting vehicle body control according to the detection result of the lighting color of the traffic light obtained by image recognition of the image around the vehicle. Therefore, it is possible to provide driving assistance to the occupant based on information with a high degree of recognition probability. When the lighting color is yellow, the accuracy of detection of the traffic light by image recognition is lower than in the case of other lighting colors. When the driving assistance device 100 detects that the lighting color of the traffic light is red, if the distance from the intersection or the traffic light to the vehicle is shorter than the distance according to the speed of the vehicle, the driving assistance device 100 immediately provides driving assistance, but when the driving assistance device 100 detects that the lighting color of the traffic light is yellow, the timing of the same driving assistance is extended until the degree of recognition probability increases. In addition, the driving assistance device 100 can accurately estimate the area of ​​the traffic light and the information of the proceeding suggestion by using at least the color of the image. In addition, image recognition recognizes whether or not an object is an object by comprehensively judging color, position, shape, etc., but if the previous image in the chronological sequence shows that the light color is yellow but is estimated to be a traffic light, the driving assistance device 100 estimates that the possibility of the object being a traffic light is high and increases the accuracy of the recognition. Also, if a change in the light color from blue to yellow is detected in the same area on the image, the driving assistance device 100 performs driving assistance because the accuracy of the recognition of the traffic light is high even if the light color is yellow.

[0065] Instead of the vehicles in the above embodiments, other moving bodies such as robots can be used.

[0066] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Recognizing at least the presence and state of a traffic light based on an image of the surroundings of the vehicle; providing driving assistance to an occupant of the vehicle based on at least a state of the traffic light; a timing for performing the driving assistance is controlled according to a lighting color of the traffic light included in the state; A driving assistance device configured as follows.

[0067] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0068] 1...vehicle system, 10...external sensor, 11...camera, 12...radar device, 16...object recognition device, 40...vehicle sensor, 80...driving operator, 100...driving support device, 120...recognition unit, 122...area identification unit, 124...progress indication estimation unit, 140...driving support processing unit, 142...notification control unit, 144...driving control unit, 160...memory unit, 200...driving drive force output device, 210...brake device, 220...steering device

Claims

1. a recognition unit that periodically acquires an image of the surroundings of the vehicle and recognizes at least the presence and state of a traffic light based on the image each time the image is acquired; A driving assistance processing unit that provides driving assistance to a vehicle occupant based on at least a state of the traffic light, The driving assistance processing unit controls a timing for performing the driving assistance in accordance with a lighting color of the traffic light included in the state, The recognition unit is a region specifying unit for specifying a traffic light recognition region which is a region of a traffic light in the image; a lighting color is detected for the traffic light recognition area of ​​the image, and when a yellow lighting color is detected, the traffic light recognition area is stored as a traffic light area candidate, and when the traffic light area candidate matches a predetermined amount or more in a plurality of chronologically consecutive images, a reliability of recognition of a traffic light in the traffic light area candidate is increased; The driving assistance processing unit performs driving assistance according to a prohibition of proceeding when a certainty of recognition of a traffic light in the traffic light area candidate is equal to or higher than a predetermined value and when the recognition unit detects a red lighting color once. Driving assistance device.

2. The driving assistance processing unit includes a notification control unit that instructs a notification device that notifies the occupant to output information about the traffic light based on the lighting color. The driving assistance device according to claim 1 .

3. The driving assistance processing unit includes a driving control unit that controls acceleration / deceleration or steering of the vehicle according to the lighting color. The driving assistance device according to claim 1 .

4. The recognition unit is A proceeding indication estimation unit that estimates a proceeding indication of the traffic light using at least color information in the traffic light recognition area and outputs the estimated indication as at least a part of the state of the traffic light. The driving assistance device according to claim 1 .

5. the driving assistance processing unit performs driving assistance based on a yellow lighting color when a yellow lighting color is detected in the traffic light recognition area of ​​the image and a green lighting color traffic light is detected in an area corresponding to the traffic light recognition area in another image obtained prior to the image; The driving assistance device according to any one of claims 1 to 4.

6. The driving assistance processing unit is performing the driving assistance when a distance between the vehicle and the traffic light is within a distance corresponding to a speed of the vehicle and the traffic light is red; When a distance between the vehicle and the traffic light is within a distance corresponding to a speed of the vehicle and the traffic light is yellow, the driving assistance is not performed. The driving assistance device according to any one of claims 1 to 4.

7. The computer a recognition step of periodically acquiring an image of an area around the vehicle and recognizing at least the presence and state of a traffic light based on the image each time the image is acquired; a driving assistance step of providing driving assistance to an occupant of a vehicle based on at least a state of the traffic light; a timing control step of controlling a timing of performing the driving assistance in accordance with a lighting color of the traffic light included in the state; In the recognition step, A process of identifying a traffic light recognition area, which is an area of ​​a traffic light in the image; a lighting color is detected for the traffic light recognition area of ​​the image, and when a yellow lighting color is detected, the traffic light recognition area is stored as a traffic light area candidate, and when the traffic light area candidate matches a predetermined level or more in a plurality of chronologically consecutive images, a process is performed to increase the likelihood of recognition of a traffic light in the traffic light area candidate; In the timing control step, when the reliability of the recognition of the traffic light in the traffic light area candidate is equal to or higher than a predetermined value, and when a red lighting color is detected once in the traffic light recognition area, driving assistance corresponding to a prohibition of proceeding is performed. Driving assistance methods.

8. On the computer, a recognition process for periodically acquiring an image of the surroundings of the vehicle and recognizing at least the presence and state of a traffic light based on the image each time the image is acquired; A driving assistance process that provides driving assistance to a vehicle occupant based on at least a state of the traffic light; a timing control process for controlling a timing for providing the driving assistance in accordance with a lighting color of the traffic light included in the state; In the recognition process, A process of identifying a traffic light recognition area, which is an area of ​​a traffic light in the image; a process of detecting a lighting color of the traffic light recognition area of ​​the image, storing the traffic light recognition area as a traffic light area candidate when a yellow lighting color is detected, and increasing the likelihood of recognition of a traffic light in the traffic light area candidate when the traffic light area candidate matches a predetermined level or more in a plurality of chronologically consecutive images; In the timing control process, when the reliability of the recognition of the traffic light in the traffic light area candidate is equal to or higher than a predetermined value, and when a red lighting color is detected once in the traffic light recognition area, driving assistance corresponding to the denial of permission to proceed is performed. program.

Images