Signal lamp recognition device, signal lamp recognition method, vehicle control device, and storage medium

Abstract

A signal light recognition device, a signal light recognition method, a vehicle control device, and a storage medium are provided, capable of accurately identifying the flashing of signal lights. The signal light recognition device includes: a camera unit that captures an image of the front of the vehicle; a signal light detection unit that detects a signal light area containing a target signal light from the image captured by the camera unit at predetermined intervals; and a flashing determination unit that determines whether the target signal light contained in the signal light area is flashing. The flashing determination unit determines whether the target signal light is flashing based on the duration of time the target signal light is continuously off (off duration) and the duration of time the target signal light is continuously on (on duration), and outputs flashing information indicating whether the target signal light is flashing or not.

Description

Technical Field

[0001] This invention relates to a traffic light recognition device, a traffic light recognition method, a vehicle control device, and a storage medium. Background Technology

[0002] Before a vehicle equipped with features such as AD (Automated Driving) or ADAS (Advanced Driver Assistance Systems) enters an intersection, it is necessary to identify flashing traffic lights in order to implement appropriate control when the traffic light corresponding to the driving lane is flashing. As such a technology, an external environment recognition device as described in Patent Document 1 is known, for example.

[0003] The vehicle exterior environment recognition device described in Patent Document 1 below determines one or more traffic lights from an image obtained by capturing the vehicle exterior environment, and derives the three-dimensional position of the determined traffic lights. It determines whether the driving road of the vehicle is a motor vehicle-only road or a general road other than a motor vehicle-only road. Based on the determined three-dimensional position of the traffic lights and specific conditions referenced according to whether the driving road is a motor vehicle-only road or a general road, it identifies the target traffic light from the determined one or more traffic lights.

[0004] Patent Document 1: Japanese Patent No. 6240475 Summary of the Invention

[0005] However, while the vehicle exterior environment recognition device described in Patent Document 1 identifies an object signal light and determines its color, it does not, for example, describe the situation where the signal light flashes at a period of 0.4 to 0.6 seconds. Therefore, the vehicle's movement may not be properly controlled when the signal light changes from a flashing state to another state.

[0006] The solutions involved in this invention were made in consideration of such circumstances, and one of their objectives is to provide a traffic light recognition device, a traffic light recognition method, a vehicle control device, and a storage medium capable of accurately identifying the flashing of traffic lights.

[0007] To address the aforementioned issues, the present invention employs the following solution.

[0008] (1): One aspect of the present invention relates to a traffic light recognition device, wherein the traffic light recognition device comprises: a camera unit that captures images of the front of the vehicle; a traffic light detection unit that detects a traffic light area containing a target traffic light from the front image captured by the camera unit at each predetermined cycle; and a flashing determination unit that determines whether the target traffic light contained in the traffic light area is flashing, wherein the flashing determination unit determines whether the target traffic light is flashing based on the duration of time the target traffic light is continuously off (i.e., the off-duration) and the duration of time the target traffic light is continuously on (i.e., the on-duration), and outputs flashing information indicating a flashing state or a non-flashing state.

[0009] (2): In the above (1) scheme, the traffic light identification device may also include a color determination unit, which determines the color state of the object traffic light in the traffic light area detected by the traffic light detection unit at each of the predetermined cycles, indicating whether the object traffic light is off or on. The flashing determination unit determines whether the object traffic light is flashing based on the color of the object traffic light immediately before it went off after it went off, i.e., the color before it went off, and the current on color determined by the color determination unit.

[0010] (3): In the above (2) scheme, the flashing determination unit may add a predetermined time elapsed amount to the extinguishing duration when the current light color state is extinguished, and change the flashing information from non-flickering state to flashing state when the current light color state is not extinguished and is the same as the light color before extinguishing, and the extinguishing duration is greater than a first time threshold equivalent to a predetermined flashing period.

[0011] (4): In the above (3) scheme, the flashing determination unit may change the flashing information from the flashing state to the non-flashing state when the flashing information indicates a flashing state and the current lighting color is not in the off state and is not the same lighting color as the light color before it was turned off.

[0012] (5): In the above (3) scheme, the flickering determination unit may change the flickering information from the flickering state to the non-flickering state when the flickering information indicates a flickering state and the extinguishing duration is less than a second time threshold that is greater than the first time threshold.

[0013] (6): In the above (3) scheme, the flickering determination unit may add the prescribed time elapsed to the lighting duration when the flickering information indicates a flickering state and the current lighting color is not in an off state but is the same as the light color before it was turned off. If the lighting duration is greater than a third time threshold that is greater than the first time threshold, the flickering information may be changed from a flickering state to a non-flickering state.

[0014] (7): In the above (3) scheme, the flashing determination unit may set the lighting duration to an initial value when the current lighting color is in an off state, and the flashing determination unit may set the off duration and the lighting duration to an initial value when the current lighting color is not in an off state and is a different light color from the light color before it was turned off.

[0015] (8): In the above scheme (3), the flashing determination unit may output the color of the light before it is turned off as the lighting color of the signal light when the flashing information is in a flashing state, and the flashing determination unit may output the current lighting color as the lighting color of the signal light when the flashing information is in a non-flashing state.

[0016] (9): Another aspect of the present invention relates to a vehicle control device, wherein the vehicle control device includes a traffic light recognition device and a reporting unit, the traffic light recognition device comprising: a camera unit that captures images of the front of the vehicle; a traffic light detection unit that detects a traffic light area containing a target traffic light from the front image captured by the camera unit at each predetermined cycle; a flashing determination unit that determines whether the target traffic light contained in the traffic light area is flashing; and a light color determination unit that determines, at each predetermined cycle, the light color state indicating whether the target traffic light contained in the traffic light area detected by the traffic light detection unit is off or on, the flashing determination unit based on the duration of the target traffic light being off (i.e., the off-duration) and the duration of the target traffic light being on (i.e., the on-duration). The system continues to determine whether the target traffic light is flashing and outputs flashing information indicating whether it is flashing or not. The flashing determination unit determines whether the target traffic light is flashing based on the light color immediately before it went out (i.e., the light color before it went out) when the light color determination unit determines that the target traffic light is flashing, and the flashing determination unit outputs flashing information indicating the flashing state, and other traffic participants are detected at the intersection corresponding to the target traffic light in the forward image captured by the camera unit, and the vehicle is about to enter the intersection, and reports to the occupants of the vehicle requesting them to stop.

[0017] (10): Another aspect of the present invention relates to a vehicle control device, wherein the vehicle control device includes a traffic light recognition device and a reporting unit, the traffic light recognition device comprising: a camera unit that captures images of the front of the vehicle; a traffic light detection unit that detects a traffic light area containing a target traffic light from the front image captured by the camera unit at each predetermined cycle; a flashing determination unit that determines whether the target traffic light contained in the traffic light area is flashing; and a light color determination unit that determines, at each predetermined cycle, the light color state indicating whether the target traffic light contained in the traffic light area detected by the traffic light detection unit is off or on, the flashing determination unit being off based on the duration the target traffic light is continuously off. The system determines whether the target traffic light is flashing based on the duration of illumination and the duration of continuous illumination of the target traffic light, i.e., the illumination duration, and outputs flashing information indicating whether it is flashing or not. The flashing determination unit determines whether the target traffic light is flashing based on the light color immediately before it went out (i.e., the light color before it went out) when the light color determination unit determines that the target traffic light's light color is the light color indicating a stop, and the flashing determination unit outputs flashing information indicating a flashing state, and the vehicle is about to enter the intersection, and reports to the occupants of the vehicle requesting them to stop.

[0018] (11): Another aspect of the present invention relates to a vehicle control device, wherein the vehicle control device includes a traffic light recognition device, the traffic light recognition device comprising: a camera unit that captures images of the front of the vehicle; a traffic light detection unit that detects a traffic light area containing a target traffic light from the front image captured by the camera unit at each predetermined cycle; a flashing determination unit that determines whether the target traffic light contained in the traffic light area is flashing; and a light color determination unit that determines, at each predetermined cycle, the light color state representing the off or on color of the target traffic light contained in the traffic light area detected by the traffic light detection unit, wherein the flashing determination unit determines whether the target traffic light is flashing based on the duration of the target traffic light being continuously off (i.e., the off-duration) and the duration of the target traffic light being continuously on (i.e., the on-duration), and outputs a table. The flashing information indicates whether the traffic light is flashing or not. The flashing determination unit determines whether the traffic light is flashing based on the color of the traffic light immediately before it went out (i.e., the color before it went out) when the traffic light went out after being lit, and the current lit color determined by the color determination unit. The vehicle control device controls the vehicle in the following manner: when the color determination unit determines that the lit color of the traffic light is a stop color, and the flashing determination unit outputs flashing information indicating a flashing state, and the vehicle is about to enter the intersection, the vehicle is temporarily stopped at the stop line of the intersection. If no other traffic participants are detected in the forward image captured by the camera unit at the intersection corresponding to the traffic light, the vehicle is allowed to enter the intersection.

[0019] (12): Another aspect of the present invention relates to a traffic light recognition method, wherein the traffic light recognition method enables a computer to perform the following processing steps: detecting a traffic light area containing a target traffic light in a predetermined period from a frontal image captured by a camera capturing the front of the vehicle; and determining whether the target traffic light contained in the traffic light area is flashing, based on the duration of the target traffic light being continuously off (i.e., the off-duration) and the duration of the target traffic light being continuously on (i.e., the on-duration), and outputting flashing information indicating a flashing state or a non-flashing state.

[0020] (13): Another aspect of the present invention relates to a storage medium that is a non-transitory storage medium storing a program and readable by a computer, wherein the program causes the computer to perform: a step of detecting a traffic light area containing a target traffic light in a predetermined period from a forward image captured by a camera capturing the front of the vehicle; and a step of determining whether the target traffic light contained in the traffic light area is flashing, based on the duration of time the target traffic light is continuously off (i.e., the off-duration) and the duration of time the target traffic light is continuously on (i.e., the on-duration), and outputting flashing information indicating a flashing state or a non-flashing state.

[0021] Based on the above schemes (1), (2), (3), (12) and (13), the flashing of traffic lights can be accurately identified.

[0022] According to the above scheme (4), it is possible to determine the scene where the object's signal light flashes and changes to another color.

[0023] According to the above scheme (5), it is possible to determine the scenario where the object signal light is flashing but no longer flashing.

[0024] According to the above scheme (6), it is possible to determine the scenario where all the lights are lit up after the object signal lights flash and then disappear from the object signal lights.

[0025] According to the above scheme (7), by initializing the extinguishing duration and the lighting duration, flickering can be accurately determined in subsequent cycles.

[0026] According to the above scheme (8), the color of the target signal light can be accurately output according to whether it is flashing or not.

[0027] According to the above scheme (9), it is possible to report and request a stop when entering an intersection where the light is flashing to indicate attention.

[0028] According to the above scheme (10), it is possible to report and request to stop when entering an intersection where the light is flashing to indicate a stop.

[0029] According to the above scheme (11), the vehicle can automatically stop temporarily at a red light flashing intersection, and after confirming the surrounding area at the intersection, the vehicle can pass through the intersection if there are no other traffic participants at the intersection. Attached Figure Description

[0030] Figure 1 This is a structural diagram of vehicle system 1 that utilizes the vehicle control device in the implementation method.

[0031] Figure 2This is a block diagram illustrating an example of the functional structure of the first control unit 110 and the second control unit 120 in the embodiment.

[0032] Figure 3 This is a block diagram illustrating an example of the internal structure of the identification unit 112 in the embodiment.

[0033] Figure 4 This is a diagram illustrating an example of the functional structure of the signal light detection unit 300 and the flashing determination unit 304 in the embodiment.

[0034] Figure 5 This is a diagram showing an outline of the processing performed by the flicker determination unit 304 in the embodiment.

[0035] Figure 6 This is a diagram illustrating an example of the functional structure of the flicker determination unit 304 in the embodiment.

[0036] Figure 7 This is a flowchart illustrating an example of the processing steps of the flicker detection module 500 in the implementation embodiment.

[0037] Figure 8 This is a flowchart illustrating an example of the processing steps of the color comparison module 502 in the implementation embodiment.

[0038] Figure 9 This is a flowchart illustrating an example of the processing steps of the extinguishing duration determination module 504 in the implementation embodiment.

[0039] Figure 10 This is a flowchart illustrating an example of the processing steps of the lighting duration determination module 506 in the implementation method.

[0040] Figure 11 This is a flowchart illustrating an example of the processing steps of the output management module 508 in the implementation embodiment. Detailed Implementation

[0041] Hereinafter, with reference to the accompanying drawings, embodiments of the traffic light recognition device, traffic light recognition method, vehicle control device, and storage medium of the present invention will be described.

[0042] (Overall structure)

[0043] Figure 1 This is a structural diagram of vehicle system 1 utilizing the vehicle control device in the embodiment. The vehicle equipped with vehicle system 1 is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or electricity discharged from a secondary battery or fuel cell.

[0044] Vehicle system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) system 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driver monitoring camera 70, operating components 80, an automatic driving control device 100, a driving force output device 90, a braking device 92, and a steering device 94. These devices and equipment are interconnected through multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. Figure 1 The structure shown is just one example; you can omit part of the structure or add other structures.

[0045] Camera 10 is a camera unit that captures images of the front of the vehicle M as it travels. Camera 10 may be a digital camera utilizing a solid-state imaging element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 can be mounted anywhere on the vehicle equipped with Vehicle System 1 (hereinafter referred to as the vehicle M). When capturing images of the front, camera 10 may be mounted on the upper part of the windshield, behind the rearview mirror inside the vehicle, etc. Ideally, camera 10 should have a defined camera direction within the range used to capture traffic lights. Camera 10 may periodically and repeatedly capture images of the perimeter of the vehicle M. Camera 10 may also be a stereo camera.

[0046] Radar device 12 radiates millimeter-wave or other radio waves around the vehicle M and detects the radio waves reflected by objects (reflected waves) to detect at least the position (distance and orientation) of the objects. Radar device 12 can be installed at any location on the vehicle M. Radar device 12 can also detect the position and speed of objects using FM-CW (Frequency Modulated Continuous Wave) method.

[0047] The LIDAR14 illuminates the periphery of the vehicle M with light (or electromagnetic waves of a wavelength close to light) and measures the scattered light. The LIDAR14 detects the distance to the object based on the time from the emission of light to the reception of light. The illuminated light can be, for example, a pulsed laser. The LIDAR14 can be mounted at any location on the vehicle M.

[0048] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the cameras 10, radar device 12, and LIDAR 14 to identify the position, type, speed, etc. of objects. The object recognition device 16 outputs the recognition results to the autonomous driving control device 100. Alternatively, the object recognition device 16 can directly output the detection results from the cameras 10, radar device 12, and LIDAR 14 to the autonomous driving control device 100. The object recognition device 16 can also be omitted from the vehicle system 1.

[0049] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using cellular networks, Wi-Fi networks, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via wireless base stations.

[0050] HMI30 provides various information to the occupants of vehicle M and accepts input operations performed by the occupants. HMI30 may include, for example, a display device 32 and operation buttons 34, and may also include a speaker, buzzer, etc.

[0051] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting angular velocity about a vertical axis, and an orientation sensor for detecting the orientation of the vehicle M.

[0052] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a path determination unit 53.

[0053] The navigation device 50 stores the first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory.

[0054] The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M can also be determined or supplemented by using the INS (Inertial Navigation System) output from the vehicle sensor 40.

[0055] The navigation HMI52 includes a display device, speakers, a touch panel, buttons, etc. The navigation HMI52 can also share some or all of the components with the aforementioned HMI30.

[0056] The route determination unit 53, for example, refers to the first map information 54 to determine the route from the position of the vehicle M determined by the GNSS receiver 51 (or any input position) to the destination input by the occupant using the navigation HMI 52 (hereinafter referred to as the map route).

[0057] The first map information 54 represents road shape information, for example, by indicating road segments and nodes connecting these segments. The first map information 54 may also include road curvature, POI (Point of Interest) information, etc. The path on the map is output to the MPU 60. The navigation device 50 can also provide route guidance using the navigation HMI 52 based on the path on the map. The navigation device 50 can also be implemented, for example, through the functions of a terminal device such as a smartphone or tablet held by the occupant. The navigation device 50 can also send its current location and destination to the navigation server via the communication device 20 and obtain a path equivalent to the path on the map from the navigation server.

[0058] MPU 60 includes, for example, a lane recommendation decision unit 61. MPU 60 stores the second map information 62 in a storage device such as an HDD or flash memory. The lane recommendation decision unit 61 divides the path on the map provided by the navigation device 50 into multiple blocks (e.g., every 100 [m] in the vehicle's direction of travel), and determines a recommended lane for each block with reference to the second map information 62. The lane recommendation decision unit 61 determines which lane to drive in from the left. When the path on the map has branching points, the lane recommendation decision unit 61 determines a recommended lane so that the vehicle M can travel on a reasonable path to the branch destination.

[0059] The second map information 62 is map information with higher precision than the first map information 54. The second map information 62 may include, for example, information about the center of a lane or the boundaries of a lane. The second map information 62 may include road information, traffic light information, traffic restriction information, residential information (address and postal code), facility information, telephone number information, and information about prohibited areas in Mode A or Mode B (described later). The second map information 62 may also include information linking traffic lights to intersections. The second map information 62 can be updated in real time by communicating with other devices via the communication device 20.

[0060] The driver monitoring camera 70 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS. The driver monitoring camera 70 can be installed anywhere in the vehicle M in a position and orientation that allows it to capture images of the head of the occupant (hereinafter referred to as the driver) seated in the driver's seat from the front (with the face facing outwards). For example, the driver monitoring camera 70 can be installed above a display device located in the center of the dashboard of the vehicle M.

[0061] The control element 80 includes, for example, a steering wheel, accelerator pedal, brake pedal, gear shift lever, and other control elements. Sensors are installed on the control element 80 to detect the amount or presence of operation, and the detection results are output to some or all of the following: the automatic driving control device 100, the driving force output device 90, the braking device 92, and the steering device 94. The control element 80 does not necessarily have to be ring-shaped; it can also be an irregularly shaped steering gear, a lever, a button, etc.

[0062] The autonomous driving control device 100 includes, for example, a first control unit 110 and a second control unit 120. The first control unit 110 and the second control unit 120 are respectively implemented by executing programs (software) using hardware processors such as CPUs (Central Processing Units). Some or all of these components can be implemented by hardware (including circuitry) such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), GPUs (Graphics Processing Units), and SOCs (System-on-Chips), or through the coordinated use of software and hardware. The program can be pre-stored in a storage device such as an HDD or flash memory (a storage device with a non-transitory storage medium) of the autonomous driving control device 100, or stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the autonomous driving control device 100 by mounting the storage medium (a non-transitory storage medium) onto the drive unit.

[0063] Figure 2 This is a block diagram illustrating an example of the functional structure of the first control unit 110 and the second control unit 120 in the embodiment. The first control unit 110, for example, includes an identification unit 112 and an action plan generation unit 114. The first control unit 110 may also be implemented using both AI (Artificial Intelligence) based functions and functions based on a pre-given model.

[0064] The identification unit 112 identifies the traffic light status, color, and flashing indicator based on information input via the camera 10. The traffic light status indicates whether the light is on or off. The traffic light color is, for example, any of red, yellow, or green. The flashing indicator indicates either a flashing state (active) or a non-flashing state (inactive).

[0065] The identification unit 112 can also identify the positions of objects surrounding the vehicle M. The identification unit 112 can also identify the positions of objects based on information input from the radar device 12 and LIDAR 14 via the object identification device 16. The position of an object is identified, for example, as its position on absolute coordinates with a representative point of the vehicle M (center of gravity, drive shaft center, etc.) as the origin, and is used for control. The position of an object can be represented by representative points such as the object's center of gravity or corners, or it can be represented by a region.

[0066] The identification unit 112 can also identify the lane (driving lane) in which the vehicle M is traveling (driving lane identification unit). For example, the identification unit 112 compares the pattern of road markings obtained from the second map information 62 (hereinafter sometimes referred to as "map road markings") with the pattern of road markings around the vehicle M identified based on the image captured by the camera 10 (hereinafter sometimes referred to as "camera road markings") within a monitoring range (comparison range), thereby identifying the driving lane. The monitoring range can be the detection limit range of the identification unit 112, or a part of the detection limit range. In the case where other vehicles are traveling in front of the vehicle M within the monitoring range, the identification unit 112 can also determine the camera road markings to be compared with the map road markings by referring to the trajectory of the other vehicles. For example, the identification unit 112 can also determine the markings along the trajectory (speed) direction of other vehicles from among a plurality of candidate markings for camera road markings as the camera road markings.

[0067] The identification unit 112 can also identify the position and orientation of the vehicle M relative to the driving lane when identifying the driving lane. For example, the identification unit 112 can identify the deviation of the reference point of the vehicle M from the center of the lane, and the angle formed by the direction of travel of the vehicle M relative to the line connecting the centers of the lanes, as the relative position and orientation of the vehicle M relative to the driving lane. Alternatively, the identification unit 112 can identify the position of the reference point of the vehicle M relative to any side end (road dividing line or road boundary) of the driving lane as the relative position of the vehicle M relative to the driving lane.

[0068] The action plan generation unit 114 generates a target trajectory for the future travel of the vehicle M, in principle, by driving in the recommended lane determined by the recommended lane determination unit 61 and avoiding approaching objects identified by the identification unit 112 (except for road markings, road signs, manholes, and other objects that can be crossed). For example, the target trajectory is represented as a track that arranges the locations (track points) that the vehicle M should reach in sequence. Track points are locations that the vehicle M should reach at predetermined travel distances (e.g., a few meters) along the route. In contrast, target speed and target acceleration are generated as part of the target trajectory at predetermined sampling times (e.g., a fraction of a second).

[0069] When generating a target track, the action plan generation unit 114 can set events for automatic driving. These events include constant speed driving events, low-speed tracking events, lane change events, branching events, merging events, and takeover events. The action plan generation unit 114 generates target tracks corresponding to the initiated events.

[0070] The second control unit 120 includes, for example, an acquisition unit 122, a speed control unit 124, and a steering control unit 126. The acquisition unit 122 acquires information about the target track (track point) generated by the action plan generation unit 114 and stores it in a memory (not shown). The speed control unit 124 controls the driving force output device 90 or the braking device 92 based on the speed elements associated with the target track stored in the memory. The steering control unit 126 controls the steering device 94 based on the curvature of the target track stored in the memory. The processing of the speed control unit 124 and the steering control unit 126 is achieved, for example, through a combination of feedforward control and feedback control. As an example, the steering control unit 126 combines feedforward control corresponding to the curvature of the road ahead of the vehicle M with feedback control based on deviation from the target track.

[0071] The driving force output device 90 outputs driving force (torque) for vehicle movement to the drive wheels. The driving force output device 90 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU (Electronic Control Unit) that controls them. The ECU controls the above structure according to information input from the second control unit 120 or from the operating unit 80.

[0072] The braking device 92 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a braking ECU. The braking ECU controls the electric motor according to information input from the second control unit 120 or from the operating element 80, so that braking torque corresponding to the braking operation is output to each wheel. The braking device 92 may include, as a backup, a mechanism for transmitting hydraulic pressure generated by the operation of the brake pedal included in the operating element 80 via the master hydraulic cylinder to the hydraulic cylinder. The braking device 92 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that controls the actuator according to information input from the second control unit 120, thereby transmitting hydraulic pressure from the master hydraulic cylinder to the hydraulic cylinder.

[0073] The steering device 94 includes, for example, a steering ECU and an electric motor. The electric motor applies force to a rack and pinion mechanism to change the orientation of the steering wheels. The steering ECU drives the electric motor to change the orientation of the steering wheels according to information input from the second control unit 120 or from the operating unit 80.

[0074] (The overall functional structure and processing of the identification unit 112)

[0075] Figure 3 This is a block diagram illustrating an example of the internal structure of the identification unit 112 in the embodiment. The identification unit 112 includes, for example, a signal light detection unit 300, a lighting information storage unit 302, and a flashing determination unit 304.

[0076] The signal light detection unit 300 receives images from the camera 10. From the images captured by the camera 10, the signal light detection unit 300 detects the area containing the target signal light at each predetermined cycle.

[0077] The lighting information storage unit 302 stores the light color before it turns off, which is the processing result of the signal light detection unit 300. The light color before it turns off is the light color immediately before the target signal light turns off after it has been in the lit state. The lighting information storage unit 302 stores the flashing flag, the off duration, and the lit duration as internal holding parameters.

[0078] The blinking determination unit 304 determines whether the target signal light within the signal light area is blinking. The blinking determination unit 304 determines whether the target signal light is blinking based on the off-duration and on-duration durations, and outputs blinking information. The off-duration duration is the time the target signal light remains off. The on-duration duration is the time the target signal light remains on. The blinking information is a blinking flag indicating whether the signal light is blinking or not.

[0079] In this embodiment, the identification unit 112 detects and tracks all traffic lights, including those corresponding to the lane in which the vehicle M is traveling and those at intersections into which the vehicle M enters. The flashing period of the traffic lights is set to a value that matches the region or country. Furthermore, the identification unit 112 detects red, yellow, and green traffic lights based on their color, but is not limited to this; it can also identify traffic lights of one or two colors.

[0080] This embodiment explains that the identification unit 112 can identify the following scenarios (1), (2) and (3).

[0081] (1) The scene where the object signal light changes from a flashing state to a different light color.

[0082] (2) The scene where the object signal light stops flashing.

[0083] (3) The scenario where the object semaphore stops lighting up after it has flashed.

[0084] (1) For example, the scene changes from flashing red to yellow. (2) For example, the scene changes from flashing red to red. (3) For example, the scene changes from flashing red to completely off.

[0085] Figure 4 This is a diagram illustrating an example of the functional structure of the signal light detection unit 300 and the flashing determination unit 304 in the embodiment.

[0086] The traffic light detection unit 300 includes, for example, a signal detection unit 400, a traffic light recognition unit 402 (light color determination unit), and a tracking processing unit 404.

[0087] The signal detection unit 400 receives the image output from the camera 10, detects traffic lights using the detection model 400a, and extracts the region containing the traffic lights using the processing unit 400b. The detection model 400a is, for example, a DNN (deep neural network) that uses images containing traffic lights as learning data. The detected image of the region containing the traffic lights is output to the traffic light recognition unit 402.

[0088] The traffic light recognition unit 402 identifies lit or flashing areas in the detection image and identifies the lit color. The traffic light recognition unit 402 functions as a color determination unit, which determines the on / off state of the target traffic light within the detected traffic light area at predetermined intervals. The traffic light recognition unit 402 uses, for example, a recognition model 402a that learns by using the extracted traffic light image as learning data. The traffic light recognition unit 402 inputs the detection image to the recognition model 402a and obtains the prediction results and color states of the flashing or lit areas output from the recognition model 402a.

[0089] The tracking processing unit 404 receives the prediction result and light color status of the flashing or lit area from the traffic light recognition unit 402, and tracks the flashing or lit area. The tracking processing unit 404 outputs the light color status as the tracking result to the flashing determination unit 304.

[0090] Figure 5 This is a diagram showing an outline of the processing performed by the flicker determination unit 304 in the embodiment.

[0091] When the flashing determination unit 304 obtains the light color state indicating that the target indicator light is not lit, it measures the extinguishing duration (step S10). The extinguishing time is the period from the moment the light stops to the moment the light starts to light up again.

[0092] When the detected extinguishing duration is in the relationship of "twice the flashing period value > extinguishing duration ≥ flashing period value", the flashing determination unit 304 sets the flashing flag to active. The flashing period value is a preset value, which is a threshold used to determine the flashing of the signal light.

[0093] When the detected extinction duration is greater than twice the flashing period value, the flashing determination unit 304 sets the flashing flag to inactive.

[0094] When the target indicator light is on, and the flashing flag is active, the flashing determination unit 304 measures the illumination duration (step S12). The illumination duration is the duration from the moment the target indicator light switches from off to on. If the illumination duration exceeds a threshold, the flashing determination unit 304 sets the flashing flag to inactive. The threshold for illumination duration is set to a value used to determine if the illumination duration is longer than the duration when the target indicator light is flashing. For example, the threshold for illumination duration is set to a time corresponding to 1.5 times the length of the flashing cycle.

[0095] When the target indicator light is lit, the blinking determination unit 304 detects a different lighting color than the previous frame (step S14) and sets the blinking flag to inactive.

[0096] The blinking determination unit 304 can be input with only the light color state indicating whether the light is lit or not. In this case, the blinking determination unit 304 may not perform the processing in S14. The blinking determination unit 304 can determine whether the target signal light is blinking based on the duration of the target signal light being continuously off (off duration) and the duration of the target signal light being continuously on (on duration), and output blinking information indicating whether it is blinking or not.

[0097] Figure 6This diagram illustrates an example of the functional structure of the blink determination unit 304 in the embodiment. The blink determination unit 304 includes, for example, a blink determination module 500, a color comparison module 502, an off-duration determination module 504, an on-duration determination module 506, and an output management module 508. The blink determination module 500, color comparison module 502, off-duration determination module 504, and on-duration determination module 506 are functional units implemented by executing programs through a computer such as a CPU.

[0098] Figure 7 This is a flowchart illustrating an example of the processing steps of the flicker detection module 500 in the implementation embodiment.

[0099] The flashing determination module 500 receives the light color status from the traffic light detection unit 300 (step S100) and determines whether the target traffic light is lit based on the traffic light status (step S102). If the current state is off (step S102: "No"), the flashing determination module 500 adds a predetermined time elapsed to the off-time duration to measure the off-time duration (step S104) and outputs the off-time duration as the processing result (step S116).

[0100] When the current state is lit (step S102: "Yes"), the blinking determination module 500 determines whether the current lit color is the same as the color of the light before it was turned off (step S108). When the current lit color is not the same as the color of the light before it was turned off (step S108: "No"), the blinking determination module 500 does not change the blinking flag and outputs the result (step S116).

[0101] If the current illuminated color is the same as the color before the light went out (step S108: "Yes"), the blinking determination module 500 determines whether the extinguishing duration is greater than or equal to a blinking cycle (step S110). If the extinguishing duration is not greater than or equal to a blinking cycle (step S110: "No"), the blinking determination module 500 outputs the blinking flag without changing it (step S116). If the extinguishing duration is greater than or equal to a blinking cycle (step S110: "Yes"), the blinking determination module 500 sets the blinking flag to active (step S112), resets the extinguishing duration (step S114), and outputs the blinking flag (step S116).

[0102] In this way, the flashing determination unit 304 can add a predetermined time elapsed to the extinguishing duration when the current light color state is off, and change the flashing information (flashing flag) from a non-flashing state (inactive) to a flashing state (active) when the current light color state is not off, is the same as the light color before extinguishing, and the extinguishing duration is greater than a first time threshold (flashing period) equivalent to a predetermined flashing period. Thus, it can accurately determine whether the target signal light is flashing.

[0103] Figure 8 This is a flowchart illustrating an example of the processing steps of the color comparison module 502 in the implementation embodiment.

[0104] The color comparison module 502 is activated when the flashing flag is active, and is input with the light color status (step S202). The color comparison module 502 determines whether the target signal light is lit based on the light color status output from the signal light detection unit 300 (step S204). If the current light color status is off (step S204: "No"), the color comparison module 502 outputs without changing the flashing flag (step S210).

[0105] When the current light color state is "on" (step S204: "Yes"), the color comparison module 502 determines whether the current illuminated color is the same as the light color before it was turned off (step S206). When the current illuminated color is not the same as the light color before it was turned off (step S206: "No"), the color comparison module 502 sets the flashing flag to inactive (step S208) and outputs the flashing flag (step S210).

[0106] If the current illuminated color is the same as the color of the light before it was turned off (step S206: "Yes"), the color comparison module 502 will not change the flashing flag and will output (step S210).

[0107] In this way, the blinking determination unit 304 can change the blinking information from a blinking state to a non-blinking state when the blinking information indicates a blinking state, and the current illuminated color is not in an off state and is not the same illuminated color as before the light was turned off. Therefore, the blinking determination unit 304 can determine scenarios where the target signal light changes from a blinking state to a different color. For example, it can determine the case where the light changes from blinking red to blinking yellow.

[0108] Figure 9 This is a flowchart illustrating an example of the processing steps of the extinguishing duration determination module 504 in the implementation embodiment.

[0109] The extinction duration determination module 504 is activated when the flashing flag is active. It receives the input extinction duration (step S300) and determines whether the extinction duration is less than twice the flashing period (first time threshold) (second time threshold) (step S302). If the extinction duration is not less than twice the flashing period (step S302: "No"), the extinction duration determination module 504 outputs the determination result (step S308).

[0110] If the extinction duration determination module 504 is less than twice the flashing period (step S302: "Yes"), it sets the flashing flag to inactive (step S304), resets the extinction duration (step S306), and outputs the flashing flag as the processing result (step S308).

[0111] In this way, the flickering determination unit 304 can change the flickering information from a flickering state to a non-flickering state when the flickering information indicates a flickering state and the extinguishing duration is less than a second time threshold that is greater than a first time threshold. Therefore, the flickering determination unit 304 can determine a scenario where the target signal light changes from a flickering state to a non-flickering state. For example, it can determine a scenario where the light changes from a flashing red state to a red light.

[0112] Figure 10 This is a flowchart illustrating an example of the processing steps of the lighting duration determination module 506 in the implementation method.

[0113] The illumination duration determination module 506 is activated when the flashing indicator is active. It receives the signal light status from the signal light detection unit 300 (step S400) and determines whether the target signal light is lit based on the light color status (step S402). If the current state is off (step S402: "No"), the illumination duration determination module 506 outputs a processing result indicating that the illumination duration is 0 (step S412).

[0114] If the current light color state is "on" (step S402: "Yes"), the illumination duration determination module 506 determines whether the current illuminated color is the same as the light color before it was turned off (step S404). If the current illuminated color is not the same as the light color before it was turned off (step S404: "No"), the illumination duration determination module 506 outputs a processing result indicating that the illumination duration is 0 (step S412).

[0115] When the current illuminated color is the same as the color before the light was turned off (step S404: "Yes"), the illumination duration determination module 506 measures the illumination duration (step S406) and determines whether the illumination duration exceeds the threshold (1.5 times the illumination cycle) (step S408). If the illumination duration exceeds the threshold (1.5 times the flashing cycle) (step S408: "Yes"), the illumination duration determination module 506 sets the flashing flag to inactive (step S410) and outputs the flashing flag (step S412).

[0116] Thus, when the flashing information indicates a flashing state, if the current state is not off and the light color is the same as before it was turned off, the flashing determination unit 304 adds a predetermined time elapsed to the illumination duration. If the illumination duration is greater than a third time threshold (1.5 times the flashing period), which is larger than the first time threshold (flashing period), the flashing information is changed from a flashing state to a non-flashing state. Therefore, the flashing determination unit 304 can determine the scenario where the target traffic light changes from a flashing state to a state where all the target traffic lights are off.

[0117] Figure 11 This is a flowchart illustrating an example of the processing steps of the output management module 508 in the implementation embodiment.

[0118] The output management module 508 receives the light color status, the light color before it was turned off, and the flashing indicator from the signal light detection unit 300 (step S500), and determines whether the target signal light is lit based on the light color status (step S502). If the current state is off (step S402: "No"), the output management module 508 resets the lighting duration (step S504) and proceeds to step S514.

[0119] If the current light color is in the lit state (step S502: "Yes"), the output management module 508 determines whether the current lit color is the same as the light color before it was turned off (step S508). If the current lit color is the same as the light color before it was turned off (step S508: "Yes"), the output management module 508 proceeds to step S514.

[0120] If the current illuminated color is not the same as the color of the light before it was turned off (step S508: "No"), the output management module 508 resets the off duration (step S510), resets the illuminated duration (step S512), and determines whether the illuminated flag is activated (step S514).

[0121] When the illumination flag is active (step S514: "Yes"), the output management module 508 outputs the color of the signal light before it went out (step S516). When the illumination flag is not active (step S514: "No"), the output management module 508 outputs the signal light status, including the flashing flag and the illumination color (step S518).

[0122] In this way, the flicker determination unit 304 sets the illumination duration to an initial value when the current illuminated color is off (step S502: "No", step S504), and sets both the off duration and the illumination duration to initial values ​​when the current illuminated color is not off (step S502: "Yes") and is a different color from the light color before it was turned off (step S508: "No"). Therefore, the flicker determination unit 304 can accurately determine flickering in subsequent cycles.

[0123] The blink determination unit 304 can output the color of the light before it was turned off as the illumination color of the signal light when the blink information is in a blinking state (step S516), and output the current illumination color as the illumination color of the signal light when the blink information is in a non-blinking state (step S518). Therefore, according to the blink determination unit 304, the color of the signal light can be accurately output according to whether the target signal light is in a blinking state or a non-blinking state.

[0124] (Vehicle control in relation to the flashing of the indicator light (yellow signal light) to show attention)

[0125] The automatic driving control unit 100 includes an action plan generation unit 114, a speed control unit 124, and a steering control unit 126, and functions as a vehicle control unit that controls the vehicle M based on the traffic light status identified by the recognition unit 112 (traffic light recognition device). In the automatic driving control unit 100, the identification unit 112 identifies the target traffic light's illumination color as an attention-indicating color and outputs flashing information indicating a flashing state. At this time, if the object recognition device 16 detects other traffic participants at an intersection corresponding to the target traffic light in the forward image captured by the camera 10, and the vehicle M is about to enter the intersection, the automatic driving control unit 100 controls the HMI 30 (reporting unit) to report to the occupants of the vehicle M requesting a stop.

[0126] Therefore, when the recognition unit 112 recognizes the flashing of the attention indicator (yellow traffic light), the automatic driving control device 100 can report the detection of other traffic participants when the vehicle M enters an intersection that has established contact with the target traffic light flashing the attention indicator.

[0127] (Vehicle control for flashing of the indicator light (red light) to indicate stop (1))

[0128] In the automatic driving control device 100, the identification unit 112 identifies the target traffic light color as the stop indicator light (red light) and outputs flashing information indicating a flashing state. At this time, when the vehicle M intends to enter an intersection, the automatic driving control device 100 controls the HMI 30 (reporting unit) to report to the occupants of the vehicle M requesting a stop. Therefore, the automatic driving control device 100 can report when the identification unit 112 detects a flashing stop indicator light and attempts to enter the intersection, ignoring a temporary stop.

[0129] (Vehicle control for the flashing of the indicator light (red light) to indicate stop (2))

[0130] In the automatic driving control device 100, the recognition unit 112 identifies the illuminated color of the target traffic light as a stop color and outputs flashing information indicating a flashing state. At this time, when the vehicle M is about to enter an intersection, the automatic driving control device 100 can temporarily stop at the stop line of the intersection and, if no other traffic participants are detected in the forward image captured by the camera 10 at the intersection corresponding to the target traffic light, control the vehicle M to enter the intersection. Thus, the automatic driving control device 100 can automatically stop the vehicle M temporarily at the intersection corresponding to the target traffic light when the red light is flashing, and proceed through the intersection after confirming that no other traffic participants are detected.

[0131] The above describes specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way. Various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A traffic light recognition device, wherein, The traffic light recognition device includes: The camera unit, which films the front of the vehicle; The traffic light detection unit detects the traffic light area containing the target traffic light from the forward image captured by the camera unit at each predetermined cycle; as well as The flashing determination unit determines whether the target signal light within the signal light area is flashing. The blinking determination unit determines whether the target signal light is blinking based on the duration of the target signal light being continuously off (i.e., the off-duration) and the duration of the target signal light being continuously on (i.e., the on-duration), and outputs blinking information indicating whether the target signal light is blinking or not.

2. The traffic light recognition device according to claim 1, wherein, The traffic light recognition device includes a light color determination unit, which determines the light color status of the target traffic light within the traffic light area detected by the traffic light detection unit, indicating whether the light is off or on, according to each predetermined cycle. The blinking determination unit determines whether the target signal light is blinking based on the light color immediately before it went out (i.e., the light color before it went out) when the target signal light went out after it went out, and the current lit color determined by the light color determination unit.

3. The traffic light recognition device according to claim 2, wherein, The flashing determination unit adds a predetermined time elapsed to the duration of the light being off when the current light color state is off. If the current light color state is not off and is the same as the light color before it was turned off, and the duration of the off state is greater than a first time threshold equivalent to a predetermined flashing period, the flashing determination unit changes the flashing information from a non-flashing state to a flashing state.

4. The traffic light recognition device according to claim 3, wherein, When the flashing information indicates a flashing state, and the current illuminated color is not in an off state and is not the same illuminated color as the light color before it was turned off, the flashing determination unit changes the flashing information from a flashing state to a non-flashing state.

5. The traffic light recognition device according to claim 3, wherein, When the flickering information indicates a flickering state and the duration of the extinguishing is less than a second time threshold that is greater than the first time threshold, the flickering determination unit changes the flickering information from a flickering state to a non-flickering state.

6. The traffic light recognition device according to claim 3, wherein, When the flashing information indicates a flashing state, and the current illuminated color is not in an off state but is the same as the color before the light was turned off, the flashing determination unit adds the predetermined time elapsed to the illumination duration, and changes the flashing information from a flashing state to a non-flashing state when the illumination duration is greater than a third time threshold that is greater than the first time threshold.

7. The traffic light recognition device according to claim 3, wherein, When the current illuminated color is off, the blinking determination unit sets the illumination duration to an initial value. When the current illuminated color is not in an off state and is a different color from the color before it was turned off, the blinking determination unit sets the off duration and the illuminated duration to initial values.

8. The traffic light recognition device according to claim 3, wherein, When the flashing information indicates a flashing state, the flashing determination unit outputs the color of the pre-extinguishing light as the illumination color of the target signal light. When the flashing information is in a non-flickering state, the flashing determination unit outputs the current illuminated color as the illuminated color of the target signal light.

9. A vehicle control device, wherein, The vehicle control device includes a traffic light recognition device and a reporting unit. The traffic light recognition device includes: The camera unit, which films the front of the vehicle; The traffic light detection unit detects the traffic light area containing the target traffic light from the forward image captured by the camera unit at each predetermined cycle; A flashing determination unit determines whether the target signal light within the signal light area is flashing; as well as The light color determination unit determines, at each predetermined cycle, the light color status of the target signal light within the signal light area detected by the signal light detection unit, whether it is off or on. The blinking determination unit determines whether the target signal light is blinking based on the duration of its continuous off state (off-duration) and the duration of its continuous on state (on-duration), and outputs blinking information indicating whether it is blinking or not. The blinking determination unit also determines whether the target signal light is blinking based on the light color immediately before it went off (pre-off-duration light color) when the target signal light went off after being on, and the current on-duration light color determined by the light color determination unit. When the light color determination unit determines that the illuminated color of the target traffic light is an indicator of attention, and the flashing determination unit outputs flashing information indicating a flashing state, and other traffic participants are detected at the intersection corresponding to the target traffic light from the forward image captured by the camera unit, and the vehicle is about to enter the intersection, the reporting unit shall report to the occupants of the vehicle requesting them to stop.

10. A vehicle control device, wherein, The vehicle control device includes a traffic light recognition device and a reporting unit. The traffic light recognition device includes: The camera unit, which films the front of the vehicle; The traffic light detection unit detects the traffic light area containing the target traffic light from the forward image captured by the camera unit at each predetermined cycle; A flashing determination unit determines whether the target signal light within the signal light area is flashing; as well as The light color determination unit determines, at each predetermined cycle, the light color status of the target signal light within the signal light area detected by the signal light detection unit, whether it is off or on. The blinking determination unit determines whether the target signal light is blinking based on the duration of its continuous off state (off-duration) and the duration of its continuous on state (on-duration), and outputs blinking information indicating whether it is blinking or not. The blinking determination unit also determines whether the target signal light is blinking based on the light color immediately before it went off (pre-off-duration light color) when the target signal light went off after being on, and the current on-duration light color determined by the light color determination unit. When the light color determination unit determines that the target traffic light is a stop-indicating light color, and the flashing determination unit outputs flashing information indicating a flashing state, and the vehicle is about to enter the intersection, the reporting unit shall report to the occupants of the vehicle requesting them to stop.

11. A vehicle control device, wherein, The vehicle control device includes a traffic light recognition device. The traffic light recognition device includes: The camera unit, which films the front of the vehicle; The traffic light detection unit detects the traffic light area containing the target traffic light from the forward image captured by the camera unit at each predetermined cycle; A flashing determination unit determines whether the target signal light within the signal light area is flashing; as well as The light color determination unit determines, at each predetermined cycle, the light color status of the target signal light within the signal light area detected by the signal light detection unit, whether it is off or on. The blinking determination unit determines whether the target signal light is blinking based on the duration of its continuous off state (off-duration) and the duration of its continuous on state (on-duration), and outputs blinking information indicating whether it is blinking or not. The blinking determination unit also determines whether the target signal light is blinking based on the light color immediately before it went off (pre-off-duration light color) when the target signal light went off after being on, and the current on-duration light color determined by the light color determination unit. The vehicle control device controls the vehicle in the following manner: when the light color determination unit determines that the illuminated color of the target traffic light is an illuminated color indicating stop, and the flashing determination unit outputs flashing information indicating flashing status, and the vehicle is about to enter the intersection, the vehicle is temporarily stopped at the stop line of the intersection; and when no other traffic participants are detected in the forward image captured by the camera unit at the intersection corresponding to the target traffic light, the vehicle is allowed to enter the intersection.

12. A traffic light recognition method, wherein, The traffic light recognition method requires the computer to perform the following processes: The steps include: detecting the area containing the target signal light from the forward image captured by the camera unit that captures images of the front of the vehicle at predetermined intervals; and... The step of determining whether the traffic light of the object within the traffic light area is flashing. Based on the duration of time the target indicator light is continuously off (off duration) and the duration of time the target indicator light is continuously on (on duration), it is determined whether the target indicator light is flashing, and flashing information indicating whether it is flashing or not is output.

13. A storage medium that is a non-transitory storage medium that stores a program and is readable by a computer, wherein, The program causes the computer to execute: The steps include: detecting the area containing the target signal light from the forward image captured by the camera unit that captures images of the front of the vehicle at predetermined intervals; and... The step of determining whether the traffic light of the object within the traffic light area is flashing. Based on the duration of time the target indicator light is continuously off (off duration) and the duration of time the target indicator light is continuously on (on duration), it is determined whether the target indicator light is flashing, and flashing information indicating whether it is flashing or not is output.

Images