Signal light recognition device, signal light recognition method, and vehicle control device

Abstract

The driving assistance (ECU 20) obtains the position of a light device whose probability of being a signal light is greater than a predetermined value, i.e., a signal light candidate, from the front camera as an observed light position. In addition, the driving assistance (ECU 20) obtains the position of a signal light existing in front of the vehicle as a map information signal light position by referring to map data near the current position of the vehicle. Furthermore, the driving assistance (ECU 20) determines that the light device is a signal light based on the difference between the signal light position on the map and the position of the observed light device, i.e., the proximity (δ), being less than a predetermined identification threshold (Dth).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on Japanese Patent Application No. 2020-134991 filed in Japan on August 7, 2020, and the contents of the basic application are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates to a technology for detecting signal lights arranged along a road using a camera. Background Art

[0004] Patent document 1 discloses a method for detecting a signal light by inferring the position of a signal light using map data and performing image recognition processing on an image area corresponding to the inferred position of the signal light. The inferred position of the signal light is calculated based on the inferred position of the vehicle and the map data. Patent document 1 also discloses a structure for searching again by sliding the image area set as the detection area from the far side, specifically, from the upper end of the image, when the signal light is not found in the image area set as the detection area of ​​the signal light.

[0005] Patent Document 1: Japanese Patent No. 6331811

[0006] For example, at night, the outer shell of a signal light may not be reflected in the image, and only the light portion that is lit may be reflected in the image. In such a case where the outer shell of the signal light is not reflected, it is difficult to distinguish the light device that is the signal light from other light devices in the image recognition process, and it is difficult to detect the signal light. Other light devices include street lights, tail lights of the vehicle ahead, headlights of the oncoming vehicle, and commercial signs that emit light.

[0007] Patent Document 1 assumes that the light device as a signal light is photographed in a state where it can be identified from other light devices. In other words, Patent Document 1 does not assume a situation where the signal light is difficult to be identified from other light devices. In Patent Document 1, when multiple light devices (in other words, light sources) are detected in the detection area, it is impossible to identify which one corresponds to the real signal light. Summary of the invention

[0008] The present invention has been made in view of the above situation, and an object of the present invention is to provide a signal light recognition device, a signal light recognition method, and a vehicle control device capable of improving the detection accuracy of a signal light.

[0009] As an example, a signal light recognition device for achieving this purpose is a signal light recognition device for recognizing signal lights existing around a vehicle, wherein the signal light recognition device comprises: a vehicle position acquisition unit for acquiring the position of the vehicle; a position on a map acquisition unit for acquiring the position of the signal light registered in the map data, that is, the position of the signal light on the map, based on the position of the vehicle acquired by the vehicle position acquisition unit; an observation position acquisition unit for acquiring position information of a lighting device detected using an image frame captured by a vehicle-mounted camera as the observed light position; and a signal light recognition unit for determining whether the lighting device is a signal light based on proximity, wherein the proximity is the difference between the signal light position on the map acquired by the position on the map acquisition unit and the observed light position acquired by the observation position acquisition unit.

[0010] In the above structure, the proximity is used as a material for determining whether the detected lighting device is a signal light. According to such a structure, even when it is difficult to determine whether the detected lighting device is a signal light, the determination accuracy can be improved. In addition, even when multiple lighting devices are detected, the lighting device equivalent to the signal light can be determined by comparing the proximity of each lighting device. Therefore, the detection accuracy of the signal light can be improved.

[0011] In addition, the signal light recognition method used to achieve the above-mentioned purpose is a signal light recognition method executed by at least one processor and used to recognize signal lights existing around a vehicle, wherein the signal light recognition method includes: a vehicle position acquisition step, acquiring the position of the vehicle; a position acquisition step on a map, based on the position of the vehicle acquired in the vehicle position acquisition step, acquiring the position of the signal light registered in the map data, that is, the position of the signal light on the map; an observation position acquisition step, acquiring the position information of the lighting device detected using the image frame captured by the vehicle-mounted camera as the observation light position; and a signal light recognition step, determining whether the lighting device is a signal light based on proximity, wherein the proximity is the difference between the signal light position on the map acquired in the position acquisition step on the map and the observation light position acquired in the observation position acquisition step.

[0012] According to the above configuration, the detection accuracy of the signal light can be improved due to the same operational effects as those of the signal light recognition device.

[0013] Moreover, a vehicle control device for achieving the above-mentioned purpose is a vehicle control device that implements vehicle control corresponding to the recognition status of traffic lights existing in the vicinity of the vehicle, wherein the vehicle control device comprises: a vehicle position acquisition unit that acquires the position of the vehicle; a position on a map acquisition unit that acquires the position of the traffic light registered in the map data, that is, the position of the traffic light on the map, based on the position of the vehicle acquired by the vehicle position acquisition unit; an observation position acquisition unit that acquires the position information of the lighting device detected using the image frame captured by the vehicle-mounted camera as the observation light position; and a traffic light recognition unit that determines whether the lighting device is a traffic light based on proximity, the proximity being the difference between the position of the traffic light on the map acquired by the position on the map acquisition unit and the observation light position acquired by the observation position acquisition unit, wherein the vehicle control device is configured to notify the user that the vehicle control is not performed when the traffic light cannot be recognized even if the remaining distance from the vehicle to the position of the traffic light on the map is less than a first specified distance.

[0014] The vehicle control device includes the structure as the signal light recognition device. Therefore, according to the structure, the detection accuracy of the signal light can be improved, and the vehicle control corresponding to the recognition state can be implemented.

[0015] In addition, the reference numerals in parentheses described in the claims indicate the correspondence relationship with specific elements described in the embodiment described later as one mode, and do not limit the technical scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a diagram for explaining the overall structure of the map cooperation system.

[0017] Figure 2 It is a block diagram showing the structure of a vehicle control system.

[0018] Figure 3 It is a block diagram showing the structure of the front camera.

[0019] Figure 4 This is a block diagram showing the structure of the driving assistance ECU.

[0020] Figure 5 This is a flowchart of the signal light passing assist process executed by the driving assist ECU.

[0021] Figure 6 This is a diagram for explaining the operation of the driving assistance ECU.

[0022] Figure 7 It is a diagram showing an adjustment example of the deemed threshold.

[0023] Figure 8 A diagram used to illustrate the interaction between a vehicle control system and a map server.

[0024] Fig. 9 This is a diagram for explaining a configuration using traffic light information transmitted from a roadside device.

[0025] Fig.10 It is a diagram showing a modified example of the system configuration. DETAILED DESCRIPTION

[0026] Please refer to the attached Figure 1 The implementation mode of the vehicle control system 1 of the present invention is described below. In addition, the following description is given by taking the area where left-hand traffic is legalized as an example. In the area where right-hand traffic is legalized, the left and right can be reversed. In addition, the green light of the traffic light in the future indicates the lighting state that allows passage, and the yellow and red lights indicate the lighting state that indicates parking. The traffic light can include a lighting device that displays an arrow. The present invention can be implemented with appropriate changes to suit the laws and customs of the area where the vehicle control system 1 is used.

[0027] <Overview of overall structure>

[0028] Figure 1 FIG. 1 is a diagram showing an example of a schematic configuration of a map cooperation system 100 including a vehicle control system 1 of the present invention. Figure 1 As shown, the map cooperation system 100 includes a vehicle control system 1 built in a vehicle Ma, and a map server 3. Figure 1 In the figure, only one vehicle Ma equipped with the vehicle control system 1 is shown, but there may be a plurality of vehicles Ma equipped with the vehicle control system 1. That is, there may be a plurality of vehicles constituting the map cooperation system 100.

[0029] The vehicle control system 1 can be mounted on a variety of vehicles Ma that can travel on the road. In addition to four-wheeled vehicles, the vehicle Ma can also be a two-wheeled vehicle, a three-wheeled vehicle, etc. A bicycle with a prime mover can also be included in the two-wheeled vehicle. The vehicle Ma can also be an owner's car owned by an individual, or a vehicle used for a car sharing service or a vehicle lending service (so-called rental car). In addition, the vehicle Ma can also be a service vehicle. Service vehicles include taxis, route buses, buses, etc. Taxis and buses can also be robot taxis without drivers.

[0030] The present vehicle hereafter refers to a vehicle Ma equipped with the vehicle control system 1. The occupant sitting in the driver's seat of the present vehicle (i.e., the driver's seat occupant) is recorded as a user. The concept of the driver's seat occupant also includes an operator who has the authority to remotely operate the present vehicle. In addition, the front and rear, left and right, and up and down directions in the following description are defined based on the present vehicle. Specifically, the front and rear direction is equivalent to the length direction of the present vehicle. The left and right direction is equivalent to the width direction of the present vehicle. The up and down direction is equivalent to the height direction of the vehicle. According to other viewpoints, the up and down direction is equivalent to a direction perpendicular to a plane parallel to the front and rear direction and the left and right direction.

[0031] The vehicle control system 1 implements wireless communication with the map server 3 to download local high-precision map data, namely, partial map data, from the map server 3 for use in driving assistance, automatic driving, and navigation.

[0032] <About map data>

[0033] Here, first, all the map data of the map server 3 are explained. The map data is equivalent to map data that represents the road structure and the position coordinates of the ground objects arranged along the road with an accuracy that can be used for automatic driving. The map data has road network data, lane network data, ground object data, static POI data and dynamic POI data. Each data is structured in layers. The road network data includes the section ID, section length, number of lanes, road attributes and connection node information (such as node ID) of each road section, as well as the node ID, position coordinates, and connection section information (such as section ID) of each road node.

[0034] The road attributes include, for example, the road name, road type, road width, lane number information indicating the number of lanes, and speed limit values. The road segment data may also include data indicating the road type, such as whether it is a dedicated road for automobiles or a general road. The dedicated road for automobiles here refers to a road where pedestrians and bicycles are prohibited from entering, such as a toll road such as an expressway. The road segment data may also have attribute information indicating whether the road is a road that allows autonomous driving.

[0035] The lane network data includes lane ID, road segment ID according to lane level, road segment length and connection node information, and node ID, location coordinates and connection segment information (such as road segment ID) of each lane node. The road segment information according to lane level in the lane network data establishes a corresponding relationship with the road segment in the road network data.

[0036] The ground object data includes road end data, road surface marking data and three-dimensional object data. The road end data includes a coordinate point group indicating the position of the road end.

[0037] Pavement marking data is data indicating the location and type of pavement markings. Pavement markings refer to spray patterns drawn on the road surface for restrictions or instructions related to road traffic. Pavement markings can be called pavement spray patterns in one embodiment. For example, lane dividing lines, crosswalks, stop lines, guide strips, safety zones, restriction arrows, etc. that indicate the boundaries of lanes are included in pavement markings. Pavement spray patterns indicating road names can also be included in pavement markings. Lines, symbols, and texts assigned to the road surface are equivalent to pavement markings. In addition, pavement markings include not only spray patterns, but also differences in the color of the road surface itself, and lines, symbols, and texts formed by road spikes, stones, etc.

[0038] The road surface marking data includes lane marking data, i.e., lane marking data, and other road surface marking data, i.e., scattered road surface marking data. The lane marking data is data on linear markings that are continuously extended along the road, whereas the scattered road surface marking data is data on road surface markings that are scattered at necessary locations, such as stop signs and speed limit signs. The scattered road surface marking data indicates the location and type of the prescribed road surface markings other than the lane markings.

[0039] The demarcation line data is data about lane demarcation lines. Lane demarcation lines also include demarcation lines realized by track fixing, road spikes and other road spikes. The demarcation line data has a demarcation line ID for each lane demarcation line, and a group of coordinate points representing the set part. The demarcation line data includes pattern information such as dotted lines, solid lines, road spikes, etc. Lane demarcation lines are also called lane markings or lane markings. The demarcation lines hereafter refer to the dividing lines of lanes. The demarcation lines can include the outer lane line, the center line (the so-called center line), etc. The road end data and the demarcation line data establish a correspondence with lane information such as the lane ID and the section ID according to the lane level. The position and shape information of the road ends and demarcation lines can also be expressed by a cubic spline curve.

[0040] The three-dimensional object data indicates the position and type of three-dimensional structures set up along the road. The three-dimensional structures set up along the road refer to, for example, traffic signs, commercial signs, poles, guardrails, curbs, trees, electric poles, traffic lights, etc. Traffic signs refer to signs that are given at least one of symbols, character strings, and patterns, and function as restriction signs, guide signs, warning signs, instruction signs, etc. Guide signs include direction signs, regional name signs indicating regional names, road name signs indicating road names, and warning signs announcing the entrances and exits of highways, service areas, etc. The shape and position of each three-dimensional structure are represented, for example, by a group of coordinate points. Part or all of the above-mentioned structures set up along the road, as well as road surface markings specified such as temporary stop lines, are used as landmarks described later. That is, the map data includes data on the location and type of the landmarks.

[0041] Static POI data refers to data about static map elements other than those mentioned above. The static map elements here refer to, for example, features that require updates within 1 to 1 week. In other words, static map elements refer to map elements whose positions and existence states do not change for a few minutes to a few hours. Static POI data include toll booths, branch points for exiting from tunnels and main lines of highways, merging points, etc. Each static POI data includes type and location information. Static POI data can be, for example, data about static map elements that are not included in the detection objects of the front camera 11 described later.

[0042] Dynamic POI data is data that indicates the location of dynamic map elements. Dynamic map elements refer to map elements whose location and existence status change with the change of time from a few minutes to a few hours. Dynamic map elements include, for example, traffic jam sections, construction sections, broken-down vehicles, fallen objects, accident sites, lane restrictions, etc. Areas with localized heavy rain, i.e., rainstorm areas, sections with frozen road surfaces, i.e., sections with snow accumulation, etc. may also be included in dynamic map elements. Dynamic POI data and static POI data may be data generated based on detection data after the server or operator performs certain processing on the map data. The map data is data that indicates the location and type of ground objects and events that affect the vehicle's driving plan.

[0043] The above-mentioned map data may also be three-dimensional map data of a point group including characteristic points of road shapes and structures. The three-dimensional map data is equivalent to map data that uses three-dimensional coordinates to represent the positions of ground objects such as road ends, lane dividing lines, and traffic signs. In addition, the three-dimensional map may also be generated by REM (Road Experience Management) based on captured images. In addition, the map data may also include a driving trajectory model. The driving trajectory model is trajectory data generated by statistically integrating the driving trajectories of multiple vehicles. The driving trajectory model is obtained by averaging the driving trajectories of each lane, for example. The driving trajectory model is equivalent to data representing the driving trajectory that serves as a reference when steering assistance is performed or when autonomous driving is performed. In addition, the steering assist function is sometimes referred to as LTA (Lane Tracing Assist) or LTC (Lane Trace Control). LTA and LTC are equivalent to vehicle control that makes the vehicle travel along the lane or the trajectory of the preceding vehicle as a driving assist.

[0044] Map data is divided into multiple blocks for management. Each block corresponds to map data of a different area. Figure 1 As shown, map data is stored in units of map pictures that divide the map included area into rectangular shapes. Map pictures are equivalent to the lower-level concept of the above-mentioned blocks. Information representing the area of ​​the real world corresponding to each local picture is assigned. The information representing the area of ​​the real world is expressed, for example, by latitude, longitude, and altitude. The length of one side of the map picture is set to several hundred meters. Each local picture is assigned a unique identifier, namely a piece ID. The map data of each block or each map picture is a part of the entire map included area, in other words, local map data. The map picture is equivalent to partial map data. The map server 3 distributes partial map data corresponding to the position of the vehicle control system 1 based on a request from the vehicle control system 1.

[0045] In addition, the size of one side of the map image is not limited to several hundred meters, but may be 1km, 2km, etc. The shape of the map image may not be rectangular. The map image may also be hexagonal, circular, etc. Each map image may also be set to partially overlap with an adjacent map image. That is, each map image may also be set to overlap with other map images near the border. The map included area may be the entire country where the vehicle is used, or a part of the area. For example, the map included area may also be an area where autonomous driving of general vehicles is allowed, or an area where autonomous driving mobile services are provided. In addition, the way map data is divided may also be determined by the data size. In other words, the map included area may also be divided and managed within a range determined by the data size. In this case, each block is set to have a data volume less than a specified value. According to this method, the data size in one distribution can be made below a certain value.

[0046] For example, the above-mentioned map data can be updated at any time by comprehensively processing the detection data uploaded from multiple vehicles. In addition, the map data processed by the map server 3 of this embodiment is a detection data map (hereinafter referred to as a PD map) generated and updated by integrating the detection data observed by multiple vehicles, but is not limited to this. The map data processed by the map server 3 can also be a high-precision map (hereinafter referred to as an HD map) generated based on the results of a fixed-point survey meter, the results of a high-precision GPS measurement, and data measured by a dedicated detection car equipped with LiDAR, etc. LiDAR is the abbreviation of Light Detection and Ranging / Laser Imaging Detection and Ranging. In addition, the map data processed by the map cooperation system 100 can also be navigation map data that is used as map data for navigation under the condition that it includes ground object data such as traffic lights and landmarks. Navigation map data is equivalent to map data that is less accurate than high-precision map data and has less information related to road shape than high-precision map data.

[0047] <Regarding the configuration of the vehicle control system 1>

[0048] like Figure 2As shown, the vehicle control system 1 has a front camera 11, a vehicle status sensor 12, a locator 13, a V2X vehicle-mounted device 14, an HMI system 15, a driving actuator 16 and a driving assistance ECU 20. In addition, the ECU in the component name is the abbreviation of Electronic Control Unit, which refers to an electronic control device. In addition, HMI is the abbreviation of HumanMachine Interface. V2X is the abbreviation of Vehicle to X (Everything), which refers to a communication technology that connects various vehicles. In addition, the "V" of V2X refers to the car as the vehicle, and "X" refers to various existences other than the vehicle, such as pedestrians, other vehicles, road equipment, networks, servers, etc.

[0049] The various devices or sensors constituting the vehicle control system 1 are connected as nodes to a communication network constructed in the vehicle, namely, an in-vehicle network Nw. The nodes connected to the in-vehicle network Nw can communicate with each other. In addition, certain devices can also be configured to communicate directly with each other without passing through the in-vehicle network Nw. Figure 2 In the embodiment, the in-vehicle network Nw is configured as a bus type, but is not limited thereto. The network topology may also be a mesh type, a star type, a ring type, etc. As the standard of the in-vehicle network Nw, for example, various standards such as Controller Area Network (CAN is a registered trademark), Ethernet (registered trademark), and FlexRay (registered trademark) can be adopted.

[0050] The front camera 11 is a camera that captures the front of the vehicle at a predetermined angle of view. The front camera 11 is disposed, for example, at the upper end of the inner side of the windshield, the front grille, the roof, etc. Figure 3As shown, the front camera 11 includes: a camera body 40 that generates an image frame; and a camera ECU 41 as an ECU that detects a specified detection object by performing recognition processing on the image frame generated by the camera body 40. The camera body 40 is a structure that includes at least an image sensor and a lens. The camera body 40 generates and outputs captured image data at a specified frame rate such as 60fps. The camera ECU 41 is implemented using an image processing chip including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The camera ECU 41 includes a recognizer 411 as a functional module. The recognizer 411 is a structure that recognizes the type of an object based on the feature vector of the image generated by the camera body 40. In the recognizer 411, for example, CNN (Convolutional Neural Network) and DNN (Deep Neural Network) to which deep learning is applied can be used.

[0051] The detection objects of the front camera 11 include, for example, moving objects such as pedestrians and other vehicles. Other vehicles also include bicycles, bicycles with prime movers, and motorcycles. In addition, the front camera 11 is configured to detect lighting devices. The lighting device here refers to a structure that emits light, such as a signal light, a vehicle headlight, a taillight, a street light, etc. The description of the lighting device can also be replaced by an external light source or a light source unit.

[0052] In addition, the front camera 11 is configured to detect predetermined ground objects. Ground objects that are detection targets of the front camera 11 include road ends, road surface markings, and structures set up along the road. Road surface markings include dividing lines, temporary stop lines, and the like. Structures set up along the road include, for example, guardrails, curbstones, trees, electric poles, traffic signs, and signal lights.

[0053] In addition, part or all of the ground objects that are detected by the front camera 11 are used as landmarks in the driving assistance ECU 20. A landmark is a ground object that can be used as a mark for determining the position of the vehicle on a map. As a landmark, for example, at least one of a signboard, a signal light, a pole, a guide plate, a temporary stop line, a dividing line, etc., which is equivalent to a traffic sign such as a restriction sign or a guide sign, can be used.

[0054] In addition, among the landmarks, features that can be used as marks for longitudinal position estimation (hereinafter, longitudinal position estimation) are referred to as landmarks for longitudinal position estimation. The longitudinal direction here corresponds to the front-rear direction of the vehicle. In addition, the longitudinal direction corresponds to the direction in which the road extends when viewed from the vehicle in a straight road section, that is, the direction in which the road extends. As landmarks for longitudinal position estimation, for example, map elements that are discretely arranged along the road, such as traffic signs such as direction signs and road surface markings such as stop lines, can be used. In addition, features that can be used as marks for lateral position estimation of the vehicle (hereinafter, lateral position estimation) are also referred to as landmarks for lateral position estimation. The lateral direction here corresponds to the width direction of the road. Landmarks for lateral position estimation refer to features that exist continuously along the road, such as road ends and dividing lines.

[0055] The landmarks mentioned hereafter mainly refer to the ground objects that can be used as marks for inferring the longitudinal position of the vehicle. Of course, as mentioned above, the concept of landmarks can include dividing lines, etc. The types of ground objects used as landmarks can be changed appropriately. As landmarks, it is preferred to use traffic signs such as direction signs, road surface markings, etc. that do not change over time and have a size that can be image-recognized even from a location more than 30 meters away. The front camera 11 only needs to be configured to detect ground objects of the type set as landmarks.

[0056] In the camera ECU 41, the image processor constituting the camera ECU 41 separates and extracts the background and the detection object from the captured image based on image information including color, brightness, contrast related to color and brightness, etc. In addition, the camera ECU 41 calculates the position and direction of various detection objects based on the image including SfM (Structure from Motion) information. The relative position (distance and direction) of the detection object relative to the vehicle can also be determined based on the size and posture (such as the degree of inclination) of the object in the image. In addition, the camera ECU 41 identifies the curvature, width, etc. of the road based on the regression curve representing the detection points of the lane demarcation lines and the road ends.

[0057] As a detection result, the camera ECU 41 outputs a signal indicating the relative position, type, moving speed, etc. of each object detected. The recognition result of the type of the detected object includes a correct solution probability value indicating the likelihood of the recognition result. For example, the correct solution probability value of the recognition result of the lighting device includes the probability of being a signal light, the probability of being a tail light, the probability of being a street light, etc. The higher the probability of being a signal light, the lower the probability of being other types. In the case where the outer shell of the signal light cannot be seen at night, even if it is assumed that the detected object is really a signal light, the probability value of the detected object being a signal light will be lower than that during the day. The probability value of each type can be a score indicating the degree of consistency of the feature quantity. The output signal of the camera ECU 41 is input to the driving assistance ECU 20 via the in-vehicle network Nw. In the future, for convenience, the probability of the type of the lighting device calculated and detected by the camera ECU 41 being a signal light is also recorded as the signal light probability P. The detection result of the front camera 11 can also be replaced by a recognition result or a recognition result.

[0058] In addition, the front camera 11 can also be configured to provide image frames for object recognition to the driving assistance ECU 20 or the like via the in-vehicle network Nw. The image frame is equivalent to the observation data of the front camera 11. The observation data is equivalent to the original data observed by the sensor, or the data before the recognition process is performed. In addition, the object recognition process based on the observation data can also be performed by an ECU outside the sensor such as the driving assistance ECU 20. For example, part of the functions of the camera ECU 41 (mainly the object recognition function) can also be possessed by the driving assistance ECU 20. In this case, the front camera 11 only needs to provide the image data as the observation data to the driving assistance ECU 20.

[0059] The vehicle state sensor 12 is a sensor group that detects state quantities related to the driving control of the vehicle. The vehicle state sensor 12 includes a vehicle speed sensor, a steering sensor, an acceleration sensor, a yaw rate sensor, and the like. The vehicle speed sensor detects the speed of the vehicle. The steering sensor detects the steering angle of the vehicle. The acceleration sensor detects accelerations such as the front and rear acceleration and the lateral acceleration of the vehicle. The yaw rate sensor detects the angular velocity of the vehicle. In addition, as the vehicle state sensor 12, the type of sensor used by the vehicle control system 1 only needs to be appropriately designed, and it is not necessary to have all of the above-mentioned sensors. In addition, the vehicle state sensor 12 can include, for example, a rainfall sensor for detecting rainfall and an illumination sensor for detecting external brightness.

[0060] The locator 13 is a device that generates high-precision position information of the vehicle by combining multiple information for composite positioning. The locator 13 is configured using a GNSS receiver, for example. A GNSS receiver is a device that sequentially detects the current position of the GNSS receiver by receiving navigation signals transmitted from positioning satellites that constitute a GNSS (Global Navigation Satellite System). For example, when a GNSS receiver is capable of receiving navigation signals from four or more positioning satellites, it outputs a positioning result every 100 milliseconds. As GNSS, GPS, GLONASS, Galileo, IRNSS, QZSS, Beidou, etc. can be used.

[0061] The locator 13 locates the position of the vehicle in sequence by combining the positioning result of the GNSS receiver and the output of the inertial sensor. For example, in a tunnel or other place where the GNSS receiver cannot receive the GNSS signal, the locator 13 uses the yaw rate and the vehicle speed to perform dead reckoning (i.e., autonomous navigation). The locator 13 can also use the output of the acceleration sensor and the gyroscope sensor for dead reckoning. The located vehicle position information is output to the in-vehicle network Nw and is used by the driving assistance ECU 20 and the like. In addition, part of the functions of the locator 13 can also be possessed by the driving assistance ECU 20.

[0062] The V2X vehicle-mounted device 14 is a device for implementing wireless communication between this vehicle and other devices. The V2X vehicle-mounted device 14 has a wide area communication unit and a narrow area communication unit as communication modules. The wide area communication unit is a communication module for implementing wireless communication based on a prescribed wide area wireless communication standard. As the wide area wireless communication standard here, for example, multiple standards such as LTE (Long Term Evolution), 4G, and 5G can be adopted. In addition, the wide area communication unit can also be configured to: in addition to communication via a wireless base station, it can also implement wireless communication with other devices directly, in other words, not via a base station, in accordance with the wide area wireless communication standard. That is, the wide area communication unit can also be configured to implement cellular V2X. This vehicle is a networked car that can be connected to the network by installing the V2X vehicle-mounted device 14. For example, the driving assistance ECU 20 can download and use high-precision map data corresponding to the current position from the map server in cooperation with the V2X vehicle-mounted device 14.

[0063] The narrow-area communication unit of the V2X vehicle-mounted device 14 is a communication module for directly implementing wireless communication with other mobile bodies and roadside devices that exist around the vehicle, based on a communication standard in which the communication distance is limited to several hundred meters, namely the narrow-area communication standard. Other mobile bodies are not limited to vehicles, but can include pedestrians, bicycles, etc. As a narrow-area communication standard, any standard such as the WAVE (Wireless Access in Vehicular Environment) standard disclosed in IEEE1609 and the DSRC (Dedicated Short Range Communications) standard can be adopted. A roadside device is a communication device that is installed outside a vehicle, such as along a road.

[0064] The HMI system 15 is a system that provides an input interface function for receiving user operations and an output interface function for presenting information to the user. The HMI system 15 includes a display 151 and an HCU (HMI Control Unit) 152. In addition, as a unit for presenting information to the user, in addition to the display 151, a speaker, a vibrator, a lighting device (such as an LED), etc. can also be used.

[0065] The display 151 is a device for displaying images. The display 151 is, for example, a so-called center display provided at the top of the center portion of the instrument panel in the vehicle width direction. The display 151 is capable of full-color display, and can be implemented using a liquid crystal display, an OLED (Organic Light Emitting Diode) display, a plasma display, or the like. In addition, the display 151 may also be a head-up display that projects a virtual image on a portion of the windshield in front of the driver's seat. In addition, the display 151 may also be an instrument display.

[0066] HCU152 is a structure that comprehensively controls information prompts to the user. HCU152 is implemented using, for example, a processor such as a CPU, a GPU, a RAM (Random Access Memory), a flash memory, and the like. HCU152 controls the display screen of the display 151 based on information provided from the driving assistance ECU20 and a signal from an input device not shown in the figure. For example, based on a request from the driving assistance ECU20, the HCU152 displays an icon image indicating the execution status of driving assistance on the display 151. In addition, the medium for information prompts to the user is not limited to the display 151. HCU152 can also be configured to use a speaker or the like to prompt information to the user.

[0067] The travel actuator 16 is an actuator for travel. The travel actuator 16 includes, for example, a brake actuator, an electronic throttle, a steering actuator, etc., which are brake devices. The steering actuator also includes an EPS (Electric Power Steering) motor. The travel actuator 16 is controlled by the driving assistance ECU 20. In addition, a steering ECU for steering control, a power unit control ECU for acceleration and deceleration control, and a brake ECU may be sandwiched between the driving assistance ECU 20 and the travel actuator.

[0068] The driving assistance ECU 20 is an ECU that assists the driving operation of the driver's seat occupant based on the detection result of the front camera 11. For example, the driving assistance ECU 20 controls the driving actuator 16 based on the detection result of the front camera 11, thereby performing part or all of the driving operation instead of the driver's seat occupant. The driving assistance ECU 20 can also be an automatic driving device that allows the vehicle to drive autonomously based on the user inputting an autonomous driving instruction.

[0069] The driving assistance ECU 20 is mainly composed of a computer including a processing unit 21, a RAM 22, a storage 23, a communication interface 24, and a bus connecting them. The processing unit 21 is hardware for calculation processing combined with the RAM 22. The processing unit 21 is a structure including at least one calculation core such as a CPU. The processing unit 21 performs various processes by accessing the RAM 22. The processing unit 21 is equivalent to a processor. The storage 23 is a structure including a non-volatile storage medium such as a flash memory. In the storage 23, a driving assistance program is stored as a program executed by the processing unit 21. The case where the processing unit 21 executes the above program is equivalent to the case where a driving assistance method including a traffic light recognition method is executed as a method corresponding to the driving assistance program. The communication interface 24 is a circuit for communicating with other devices via the in-vehicle network Nw. The communication interface 24 can be implemented using analog circuit elements, ICs, etc. The driving assistance ECU 20 is equivalent to a traffic light recognition device and a vehicle control device.

[0070] <About driving assistance ECU 20>

[0071] Here, use Figure 4 The functions and operations of the driving assistance ECU 20 are described. The driving assistance ECU 20 provides the driving assistance program stored in the memory 23 by the processing unit 21. Figure 4 The functions corresponding to the various functional modules shown in the figure. That is, in the driving assistance ECU 20, as functional modules, there are a tentative position acquisition unit F1, a map acquisition unit F2, a camera output acquisition unit F3, a vehicle state acquisition unit F4, a detailed position calculation unit F5, a position error evaluation unit F6, an environment recognition unit F7, a control planning unit F8, and a control execution unit F9.

[0072] The provisional position acquisition unit F1 acquires the position information of the host vehicle from the positioner 13. The provisional position acquisition unit F1 performs dead reckoning based on the output of the yaw rate sensor, etc., with the position calculated by the detailed position calculation unit F5 described later as the starting point. The provisional position acquisition unit F1 may also have the function of the positioner 13.

[0073] The map acquisition unit F2 acquires map data corresponding to the current position of the vehicle by wirelessly communicating with the map server 3 via the V2X vehicle-mounted device 14. For example, the map acquisition unit F2 requests the map server 3 and acquires partial map data related to the roads that the vehicle is scheduled to pass within a specified time. The partial map data acquired from the map server 3 is stored in the map storage unit M1, for example. The map storage unit M1 is implemented using, for example, a part of the storage area provided by the storage 23 or the RAM 22. The map storage unit M1 is implemented using a non-migrating physical storage medium.

[0074] The current position used when acquiring a map can be a position determined by the provisional position acquisition unit F1 or the detailed position calculation unit F5. For example, when the detailed position calculation unit F5 can calculate the current position, the map data is acquired using the position information. On the other hand, when the detailed position calculation unit F5 cannot calculate the current position, the map data is acquired using the position coordinates acquired by the provisional position acquisition unit. The map data includes the location of the traffic light as described above. Therefore, the map acquisition unit F2 is equivalent to a structure that acquires the position coordinates of the traffic light registered on the map, that is, the location of the traffic light on the map, that is, the location acquisition unit on the map.

[0075] In addition, the vehicle control system 1 may also include an on-board storage device storing map data. In this case, the map acquisition unit F2 may be configured to sequentially read map data of the surrounding area of ​​the current position from the on-board storage device. That is, the map data acquired by the map acquisition unit F2 may also be stored locally in the vehicle or in the cloud.

[0076] The camera output acquisition unit F3 acquires the recognition result of the front camera 11 for the lighting device, other moving objects, landmarks, road ends, lane demarcation lines, etc. For example, the camera output acquisition unit F3 acquires the recognition result of the lighting device including the correct answer probability value for each position and type. Specifically, as the recognition result of the lighting device, the position of the lighting device detected by the front camera 11 and the signal light probability P are acquired. The detection position of the lighting device acquired by the camera output acquisition unit F3, i.e., the observed lighting position, is equivalent to the position of the signal light observed by the front camera 11, i.e., the observed signal light position. The camera output acquisition unit F3 is equivalent to the observed position acquisition unit.

[0077] In addition, the camera output acquisition unit F3 acquires the position, moving speed, type, size, etc. of other moving objects. Other moving objects include other vehicles such as cars, pedestrians, bicycles, etc. Furthermore, the camera output acquisition unit F3 acquires the relative position, type, etc. of the landmark from the front camera 11.

[0078] In addition, the camera output acquisition unit F3 transforms the relative position coordinates of the landmark obtained from the camera ECU 41 into position coordinates in the global coordinate system (hereinafter also recorded as observation coordinates). For example, the observation coordinates of the landmark are calculated by combining the current position coordinates of the vehicle and the relative position information of the ground object relative to the vehicle. In addition, the camera ECU 41 may also implement the calculation of the observation coordinates of the landmark using the current position coordinates of the vehicle. In addition, similarly to the landmark, the camera output acquisition unit F3 may transform the relative position information of the dividing line, etc. into position coordinates in the global coordinate system. The data acquired by the camera output acquisition unit F3 is provided to the environment recognition unit F7.

[0079] The vehicle state acquisition unit F4 acquires the running speed, the traveling direction, the time information, the weather, the illumination outside the vehicle compartment, the operating speed of the wipers, and the like from the vehicle state sensor 12 and the like via the in-vehicle network Nw.

[0080] The detailed position calculation unit F5 performs positioning processing based on the landmark information and route information acquired by the camera output acquisition unit F3. Positioning processing refers to the processing of determining the detailed position of the vehicle by comparing the position of the landmark or the like determined based on the image captured by the front camera 11 with the position coordinates of the ground object registered in the high-precision map data. For example, the positioning processing may include at least one of a longitudinal position estimation processing of estimating the longitudinal position using landmarks such as direction signs and a lateral position estimation processing of estimating the lateral position of the road using the observed coordinates of lane dividing lines and road ends.

[0081] As a longitudinal position inference process, the detailed position calculation unit F5 establishes a correspondence between the landmark registered on the map and the landmark observed by the front camera 11 based on the observed coordinates of the landmark. For example, the landmark registered on the map that is closest to the observed coordinates of the landmark is inferred to be the same landmark. In addition, it is preferred to use landmarks with a higher degree of feature consistency when comparing landmarks, such as using feature quantities such as shape, size, and color. If the correspondence between the observed landmark and the landmark on the map is established, the longitudinal position of the vehicle on the map will be set at a position that is longitudinally offset from the position of the landmark on the map corresponding to the observed landmark by the distance between the observed landmark and the vehicle.

[0082] For example, when the distance to the direction sign existing in front of the vehicle is determined to be 40m as a result of image recognition, the vehicle is determined to be present at a position offset 40m to the rear of the vehicle from the position coordinates of the direction sign registered in the map data. Longitudinal position estimation is equivalent to the process of determining the position of the vehicle in the direction in which the road extends. Longitudinal position estimation can also be called longitudinal positioning processing. By performing such longitudinal position estimation, the detailed remaining distance to a feature point on the road such as an intersection, a turning entrance / exit, a tunnel entrance / exit, the end of a traffic jam, in other words, a POI is determined.

[0083] In addition, when the detailed position calculation unit F5 detects multiple landmarks (such as direction signs) in front of the vehicle, the landmark closest to the vehicle among the multiple landmarks is used to estimate the longitudinal position. Regarding the recognition accuracy of the type and distance of the object based on the image, the closer the object is to the vehicle, the higher the recognition accuracy. That is, when multiple landmarks are detected, the structure of using the landmark closest to the vehicle to estimate the longitudinal position can improve the estimation accuracy of the position.

[0084] In addition, as a more preferred mode, the detailed position calculation unit F5 of this embodiment is configured to perform longitudinal position estimation processing using landmarks existing within a specified reference distance, for example, in order to ensure the accuracy of position calculation. The reference distance may be, for example, 40m. The reference distance may also be 30m or 50m. In addition, when the front camera 11 is equipped with a telephoto camera, the reference distance may also be 100m or the like. The reference distance may be set based on the performance and specifications of the front camera 11, such as the recognizable distance and viewing angle.

[0085] In addition, as a lateral position estimation process, the detailed position calculation unit F5 determines the lateral position of the vehicle relative to the road based on the distance from the left and right road ends / dividing lines recognized by the front camera 11. For example, when it is determined as a result of image analysis that the distance from the left road end to the center of the vehicle is 1.75m, it is determined that the vehicle exists at a position offset 1.75m to the right from the coordinates of the left road end. The lateral position estimation can also be called a lateral positioning process. In addition, the detailed position calculation unit F5 can also determine the identifier of the lane in which the vehicle is traveling, that is, the driving lane ID, based on the distance from the left and right road ends / dividing lines recognized by the front camera 11. The driving lane ID indicates, for example, the number of lanes on which the vehicle is traveling from the left or right road end. In addition, the detailed position calculation unit F5 can also determine the detailed position of the vehicle in the driving lane, such as the offset from the center of the lane to the left and right directions, based on the distance from the left and right dividing lines recognized by the front camera 11.

[0086] The vehicle position as a result of the positioning process can be expressed by the same coordinate system as the map data, for example, latitude, longitude, and altitude. The vehicle position information can be expressed by an arbitrary absolute coordinate system such as WGS84 (World Geodetic System 1984).

[0087] In addition, as an example, the detailed position calculation unit F5 of this embodiment is configured to use the traffic lights within the reference distance as landmarks to implement positioning processing during the day, while not using the traffic lights as landmarks at night. That is, at night, landmarks other than traffic lights are used to implement positioning processing. This is because the visibility of the housing of the traffic light deteriorates at night, and the reliability of whether it is a traffic light, in other words, the recognition accuracy deteriorates compared to the daytime. Of course, as another method, the detailed position calculation unit F5 can also be configured to use traffic lights as landmarks for positioning processing at night. As landmarks other than traffic lights, for example, road surface displays such as temporary stop lines and crosswalks, guide signs, etc. can be used. In addition, demarcation lines and road ends can be used for lateral position inference processing regardless of day or night. Therefore, the same ground features as during the day can be used to implement lateral positioning processing at night.

[0088] The detailed position calculation unit F5 performs positioning processing in sequence with a prescribed position estimation cycle. The default value of the position estimation cycle may also be 200 milliseconds or 400 milliseconds. For example, as long as the detailed position calculation unit F5 can identify (in other words, capture) a landmark, it will sequentially perform longitudinal position estimation processing with the above-mentioned position estimation cycle. When the detailed position calculation unit F5 cannot identify a landmark, as long as it can identify (in other words, capture) at least one of the dividing line and the road end, it will sequentially perform lateral position estimation processing with the position estimation cycle. At least one of the detailed position calculation unit F5 and the provisional position acquisition unit F1 is equivalent to the vehicle position acquisition unit. The vehicle position information calculated by the detailed position calculation unit F5 is provided to the provisional position acquisition unit F1, the position error evaluation unit F6, the environment recognition unit F7, etc.

[0089] Each time the detailed position calculation unit F5 performs positioning processing, the position error evaluation unit F6 calculates the difference between the current position output as the result of the positioning processing performed this time and the position calculated by the tentative position acquisition unit F1 through dead reckoning or the like, that is, the error evaluation value ρ. For example, when the positioning processing is performed using a landmark different from the landmark used last time, the position error evaluation unit F6 calculates the difference between the vehicle position coordinates calculated by the tentative position acquisition unit F1 and the result of the positioning processing as the error evaluation value ρ. As long as the positioning processing can be performed sequentially, the result of the positioning processing at the previous moment will be reflected in the tentative position information calculated by the tentative position acquisition unit F1, so the error evaluation value ρ is a sufficiently small value. For example, as long as the positioning processing can be performed sequentially, the error evaluation value ρ is suppressed to less than 0.5 m.

[0090] The error evaluation value ρ has a tendency to become larger as the period during which positioning is impossible is longer. During the period during which positioning processing cannot be performed, a provisional error evaluation value ρ is calculated by multiplying the elapsed time or the travel distance from the moment when positioning processing can be last performed by a prescribed error estimation coefficient. The position error evaluation unit F6 is equivalent to a structure that calculates the error evaluation value ρ based on the elapsed time since the last positioning was implemented, the vehicle position coordinates obtained by the positioning processing, and the degree of deviation of the vehicle position coordinates inferred by methods other than positioning. Methods other than positioning refer to positioning based on GNSS, dead reckoning using inertial sensors, and the like. The error evaluation value ρ calculated by the position error evaluation unit F6 is provided to the environment recognition unit F7. The position error evaluation unit F6 can also be built into the detailed position calculation unit F5. The position error evaluation unit F6 is equivalent to the position error acquisition unit. Since the error evaluation value ρ is a parameter indicating the magnitude of the error in the vehicle position acquired by the provisional position acquisition unit F1, it can also be called a vehicle position error.

[0091] The environment recognition unit F7 recognizes the surrounding environment of the vehicle, i.e., the surrounding environment, based on the recognition result of the front camera 11 acquired by the camera output acquisition unit F3. The surrounding environment here includes the current position of the vehicle, the driving lane, the type of road, the speed limit, the relative position of the traffic light, etc. In addition, the surrounding environment may also include the position, moving speed, shape and size of surrounding objects of other moving bodies. As a sub-function, the environment recognition unit F7 has a traffic light recognition unit F71.

[0092] The signal light recognition unit F71 is a structure that uses map data to determine whether the light device detected by the front camera 11 is a signal light. The signal light recognition unit F71 not only determines whether the light device detected by the front camera 11 is a signal light, but also includes a function of recognizing the lighting state of the light device when it is determined that the light device is a signal light. The details of the signal light recognition unit F71 will be described later.

[0093] In addition, the environment recognition unit F7 may also obtain detection results from each of the plurality of surrounding monitoring sensors and combine them to recognize the position and type of objects existing around the vehicle. In addition, the environment recognition unit F7 may also determine the surrounding environment using other vehicle information received from other vehicles by the V2X vehicle-mounted device 14, traffic information received from the roadside device through road-to-vehicle communication, and the like. The traffic information that can be obtained from the roadside device may include road construction information, traffic restriction information, traffic congestion information, weather information, speed limit, and the like.

[0094] The control plan unit F8 uses the driving environment and map data identified by the environment recognition unit F7 to generate a vehicle control plan for assisting the user's driving operation. For example, when the control plan unit F8 confirms that there is a traffic light in front of the vehicle, it makes a vehicle control plan corresponding to the lighting state of the traffic light. For example, when the traffic light is red when the vehicle reaches 100 meters before the traffic light, a deceleration driving plan is made so that the vehicle stops before the specified distance of the traffic light. The parking position may also be the position of the temporary stop line shown in the map data. In addition, if there is a vehicle ahead, the control plan can be updated at any time so that the vehicle stops behind the specified distance of the vehicle ahead. When the traffic light is in a lighting state that allows the vehicle to enter and pass through the intersection, a control plan for passing through the intersection is formulated. The lighting state that allows entry and passing through the intersection refers to the arrow signal and the green light.

[0095] The content of the control plan of the traffic light can be generated based on the predicted result of the lighting state of the traffic light before the vehicle reaches a specified distance (for example, 100m, 50m) from the traffic light. For convenience, the vehicle control that assists in driving through a road with a traffic light is called traffic light passing assistance. Traffic light passing assistance includes brake control for parking in front of the traffic light. In addition, traffic light passing assistance can also be a process of notifying the user of the existence of the traffic light and the lighting state of the traffic light in cooperation with the HMI system 15. The control plan of traffic light passing assistance can be generated and corrected not only based on the lighting state of the traffic light for the vehicle, but also based on the lighting state of the traffic light for pedestrians. The control plan of traffic light passing assistance can also be updated at any time based on the change of the lighting state of the traffic light.

[0096] In addition, the control plan unit F8 may also perform a control plan including a control schedule for the steering amount for driving in the center of the identified driving lane, or generate a path along the identified behavior or driving trajectory of the leading vehicle as a driving plan. In the case where the driving path of the host vehicle is equivalent to a road with multiple lanes on one side, the control plan unit F8 may also generate a plan candidate for changing the lane to an adjacent lane in the same direction as the driving lane of the host vehicle. In the case where it is confirmed that there is an obstacle in front of the host vehicle based on the detection results of the front camera 11 or the map data, the control plan unit F8 may also generate a driving plan to pass the side of the obstacle. The driving plan may include schedule information for acceleration and deceleration for speed adjustment in the calculated path. The control plan unit F8 may also be used to enable the host vehicle to drive autonomously.

[0097] The control execution unit F9 is a structure that outputs a control signal corresponding to the control plan determined by the control plan unit F8 to the travel actuator 16 or the HCU 152 as the control object. For example, in the case of a predetermined deceleration, a control signal for realizing the planned deceleration is output to the brake actuator and the electronic throttle. In addition, a control signal for outputting an image or voice indicating the execution state of the traffic light passing assistance is output to the HCU 152.

[0098] <About the operation flow of the driving assistance ECU 20>

[0099] Then use Figure 5 The flowchart shown in FIG. 1 illustrates the signal light auxiliary processing executed by the driving assistance ECU 20. For example, the signal light auxiliary processing is executed at a predetermined period (for example, every 200 milliseconds) while the vehicle is powered on. Figure 5 The driving power source is, for example, an ignition power source in an engine vehicle. In an electric vehicle, a system main relay corresponds to the driving power source. In this embodiment, as an example, the signal light auxiliary process includes steps S101 to S112.

[0100] In addition, in the detailed position calculation unit F5, Figure 5 The driving assistance ECU 20 sequentially performs positioning processing independently, in other words, in parallel, as shown in the flowchart. Specifically, the detailed position calculation unit F5 sequentially performs longitudinal position estimation processing using landmarks other than traffic lights. The lateral position is sequentially calculated based on at least one of the dividing line and the road end. In addition, the longitudinal position when landmarks other than traffic lights cannot be identified can be estimated based on, for example, the vehicle speed or the wheel speed. By executing the positioning processing, the detailed position of the vehicle on the map is determined.

[0101] First, in step S101, at least one of the detailed position calculation unit F5 and the provisional position acquisition unit F1 calculates or acquires the current position of the host vehicle, and the process proceeds to step S102. Step S101 corresponds to a vehicle position acquisition step.

[0102] In step S102, the environment recognition unit F7 refers to the map data acquired by the map acquisition unit F2, determines the position of the front signal light, and calculates the distance to the front signal light, that is, the remaining distance Rd, and moves to step S103. The front signal light refers to the signal light that is located closest to the vehicle among the signal lights registered in the map data and exists in front of the vehicle and faces the vehicle. The signal light facing the vehicle refers to the signal light that the vehicle should see, in other words, the signal light that the vehicle should follow. The signal light facing the oncoming vehicle and the signal light facing the crossing vehicle are not equivalent to the front signal light. In addition, the crossing vehicle refers to a vehicle traveling on other roads connected to the road where the vehicle is traveling. For example, a vehicle coming from the side at an intersection is equivalent to a crossing vehicle.

[0103] In addition, if it is confirmed that there is no traffic light within a predetermined distance (e.g., 250 m) in front of the vehicle as a result of referring to the map data in step S102, the subsequent processing may be omitted and the present flow may be terminated. When the present flow is terminated, the process is executed again from step S101 after a predetermined time, such as 200 milliseconds. Step S102 is equivalent to a step of obtaining a position on a map.

[0104] In step S103, the camera output acquisition unit F3 acquires the recognition result of the lighting device that may be a traffic light from the front camera 11 and moves to step S104. Step S103 is equivalent to the observation position acquisition step. The lighting device that may be a traffic light refers to, for example, a lighting device whose probability P of a traffic light is greater than a specified candidate threshold. The candidate threshold is a threshold used to distinguish between objects that may be traffic lights and objects that may not be traffic lights, and the specific value can be changed appropriately. The candidate threshold can be, for example, 40%. The candidate threshold can also be 50%, 75%, etc. If the lighting device that may be a traffic light is not detected in step S103, move to step S110.

[0105] In step S104, the signal light recognition unit F71 obtains the error evaluation value ρ calculated in the above manner from the position error evaluation unit F6 and moves to step S105. In step S105, the signal light recognition unit F71 determines whether the error evaluation value ρ obtained in step S104 is less than a prescribed error threshold value Pth. The error threshold value Pth may be, for example, 0.5 m, 1 m, etc. By making the error threshold value Pth less than 1 m, the possibility of exceeding the temporary stop line when the signal stops can be reduced. When the error evaluation value ρ is less than the error threshold value Pth, a positive judgment is made in step S105 and the process moves to step S106. On the other hand, when the error evaluation value ρ is greater than the error threshold value Pth, a negative judgment is made in step S105 and the process moves to step S110.

[0106] In step S106, the signal light recognition unit F71 calculates the difference (in other words, the distance) between the position of the front signal light shown in the map data and the position of the light device recognized by the front camera 11 acquired in step S102, that is, the proximity δ, and moves to step S107. The proximity δ can be a straight-line distance in three-dimensional space. In addition, the proximity δ can also be a lateral distance. The proximity δ of the lateral distance and the vertical distance can also be calculated.

[0107] In step S107, the traffic light recognition unit F71 determines whether the proximity δ calculated in step S106 is less than a prescribed identification threshold Dth. The identification threshold Dth is a threshold used to distinguish whether the lighting device that may be a traffic light extracted in step S103, i.e., the traffic light candidate, is actually a traffic light. The identification threshold Dth may be, for example, 1.0 m. The identification threshold Dth may be the same as the error threshold Pth, or a value greater than the error threshold Pth by a prescribed amount. The identification threshold Dth may also be 0.5 m, 1.5 m, etc. When the proximity δ is less than the identification threshold Dth, step S107 is affirmatively determined and the process proceeds to step S108. On the other hand, when the proximity δ is greater than the identification threshold Dth, step S107 is negatively determined and the process proceeds to step S110. Step S107 is equivalent to a traffic light recognition step.

[0108] In step S108, the signal light recognition unit F71 determines the signal light candidate extracted in step S102 as a signal light and moves to step S109. In addition, in the case where it is assumed that a plurality of signal light candidates are extracted in step S102, the proximity δ of each of the plurality of signal light candidates is calculated in step S106. Moreover, in step S108, it can be determined whether the proximity δ is less than the identification threshold value Dth for the signal light candidate with the smallest proximity δ among the plurality of signal light candidates. The signal light recognition unit F71 can determine the signal light candidate with the smallest proximity δ and the proximity δ less than the identification threshold value Dth among the plurality of signal light candidates as a signal light. In addition, the signal light recognition unit F71 can score the likelihood of being a signal light based on both the proximity δ and the signal light probability P output by the front camera 11, and determine whether it is a signal light based on the score.

[0109] In step S109 , the control planning unit F8 plans vehicle control corresponding to the remaining distance Rd to the front traffic light and the lighting state of the front traffic light recognized by the front camera 11 as traffic light passing assistance, and the control execution unit F9 executes control corresponding to the control plan.

[0110] In step S110, it is determined whether the remaining distance Rd is less than the prescribed auxiliary stop distance Rth. The auxiliary stop distance Rth is a threshold value for abandoning the execution of the traffic light passing assistance. The auxiliary stop distance Rth is equivalent to the first prescribed distance. The auxiliary stop distance Rth may also be called the limit distance. The auxiliary stop distance Rth may be, for example, 50m. The auxiliary stop distance Rth may also be 100m, 70m, 40m, etc. The auxiliary stop distance Rth is set to a distance at which the vehicle can stop at a prescribed deceleration until the vehicle reaches the temporary stop line arranged in front of the traffic light when the lighting state of the traffic light is red. The deceleration assumed here is, for example, 0.3G≒3m / sec^2, etc.

[0111] The assistance stop distance Rth may also be dynamically determined based on the driving speed of the vehicle acquired by the vehicle state acquisition unit F4. The greater the driving speed, the greater the assistance stop distance Rth is set. In other words, when the driving speed is lower than a specified value, the assistance stop distance Rth can be set to a value shorter than a specified default value. The assistance stop distance Rth may also be specified by time. For example, the assistance stop distance Rth may also be dynamically determined to be a distance equivalent to 6 seconds. When the remaining distance Rd is less than the assistance stop distance Rth, step S110 is positively determined and the process moves to step S112. On the other hand, when the remaining distance Rd is greater than the assistance stop distance Rth, step S110 is negatively determined and the process moves to step S111.

[0112] In step S111, the judgment of whether the signal light candidate is a signal light is retained, and the present process is terminated. In other words, this step is equivalent to a step of retaining the judgment of whether to interrupt the signal light through assistance. When the present process is terminated while retaining the judgment, the process is executed again from step S101 after a predetermined time such as 200 milliseconds.

[0113] In step S112, the control planning unit F8 decides to interrupt the signal light passing assistance for the signal light ahead. In addition, the control execution unit F9 cooperates with the HMI system 15 to perform processing for notifying the user that the signal light passing assistance will not be performed, thereby terminating the present process. For example, in step S112, an instruction signal for causing the display 151 to display a text message or an icon image indicating that the signal light passing assistance will not be performed is output to the HCU 152, and is displayed. In step S112, a voice message indicating that the signal light passing assistance will not be performed may also be output from the speaker. In addition, an icon image indicating that the signal light passing assistance will not be performed may also be displayed on the display 151. In addition, the content notified in step S112 may also be that the signal light cannot be recognized. The content notified in step S112 may be a variety of information related to the recognition status of the signal light. In addition, in step S112, an error signal indicating that the detection of the signal light has failed may also be uploaded to the map server 3 via the V2X vehicle-mounted device 14. According to this structure, the map server 3 is able to determine the location and time period where it is difficult to recognize the signal light. Preferably, the error signal includes the current position, identification information of the front signal light, model information of the front camera 11, and the like.

[0114] In the above structure, proximity δ is used as a material for determining whether the detected light device is a traffic light. According to this structure, since the matching between the position information registered in the map and the observation result is taken into consideration, the accuracy of determination can be improved even when it is difficult to determine whether the detected light device is a traffic light only by the feature quantity of the image. In addition, even when multiple light devices are detected, the light device equivalent to the traffic light can be determined by comparing the proximity δ of each light device. For example, the light device with the smallest proximity δ can be determined as the light of the traffic light.

[0115] In addition, it is difficult to judge whether the detected light device is a signal light, for example, when the outer shell of the signal light is assimilated with the background at night. In other words, the assimilation with the background includes a state where the outline of the outer shell has no brightness difference with the background that can detect the edge. In addition, due to rain, backlight, etc., there is less image information for identifying it as a signal light, which also includes the case where it is difficult to judge whether the detected light device is a signal light. The information for identifying it as a signal light refers to the outline shape, color information, etc. of the outer shell. When it rains, the shape of the object in the image is deformed due to raindrops attached to the windshield and the reflector in front of the camera, so the feature quantity for identifying the signal light as a signal light is reduced. In backlight, the edge part of the outer shell, such as the number of light parts, which represents the detailed structure in the outer shell, becomes unclear, or the color information is reduced, so the feature quantity for identifying the signal light as a signal light is reduced. In addition, the structure of the present invention can improve the judgment accuracy of whether the detected light device is a signal light in various scenes, such as when turning.

[0116] In addition, the signal light recognition unit F71 of the present invention implements signal light determination using the proximity δ, on the condition that the error evaluation value ρ is less than the error threshold value Pth. According to this structure, the possibility of incorrectly determining the signal light can be further reduced. And, as in the above-mentioned structure, the structure of determining whether the detected lighting device is a signal light based on the proximity δ can reduce the possibility of incorrectly determining the image of the signal light projected on the window glass of a building, etc. as an actual signal light. This is because it can be expected that the image of the signal light projected on the window glass of a building, etc. by reflection of light is significantly away from the setting position of the signal light. In addition, the setting position of the signal light is registered in the PD map based on the detection data uploaded from the vehicle. Therefore, the setting position of the signal light registered in the PD map may include a certain degree of error. In view of such a situation, by making the error threshold value Pth have a certain degree of width, the possibility of incorrectly determining the signal light itself as not being a signal light can be reduced.

[0117] In addition, in the structure disclosed in Patent Document 1 and the like, when turning left or right at an intersection, the traffic light facing the intersecting vehicle may be mistakenly recognized as the traffic light facing the own vehicle as the vehicle body rotates. In response to such concerns, according to the structure of the present invention, the traffic light that the own vehicle should see is extracted from the map, and the traffic light is recognized based on the proximity δ to the traffic light. Of course, it is assumed that the traffic light facing the intersecting vehicle is sufficiently separated from the traffic light that the own vehicle should see, for example, by more than 5 meters. Therefore, according to the structure of the present invention, the possibility of mistakenly recognizing the traffic light facing the intersecting vehicle as the traffic light facing the own vehicle and performing unexpected actions such as parking at the intersection can be reduced.

[0118] In addition, when a light device with a sufficiently high probability of being a traffic light is detected, the process of determining whether the detected light device is a traffic light using the proximity δ may be omitted. For example, when a light device with a traffic light probability P of a predetermined value (e.g., 95%) or more is detected, the determination process using the proximity δ may be omitted. According to this configuration, the processing load of the processing unit 21 can be reduced. This configuration is equivalent to a configuration in which, when a light device with a traffic light probability P of a predetermined value or more is detected, the light device is regarded as a traffic light, and, on the other hand, when a light device with a traffic light probability P of a predetermined value or more is not detected, a light device equivalent to a traffic light is searched based on the proximity δ. In addition, in order to suppress the erroneous recognition of a traffic light for an intersecting vehicle as a traffic light for the vehicle itself, the recognition process of the lighting state of the traffic light may be canceled in a state where the yaw rate is greater than a predetermined threshold value within the intersection. In addition, the traffic light recognition unit F71 may be configured to recognize a traffic light for an intersecting vehicle and a traffic light for an oncoming vehicle, in addition to the traffic light in front. The determination process using the proximity δ may also be applied when recognizing a traffic light for an intersecting vehicle and a traffic light for an oncoming vehicle. Traffic lights facing crossing vehicles and traffic lights facing oncoming vehicles can be called traffic lights facing other vehicles.

[0119] The embodiments of the present invention are described above, but the present invention is not limited to the above-mentioned embodiments. The various structures described later are also included in the technical scope of the present invention, and in addition to the following, various changes can be made to implement within the scope without departing from the main purpose. For example, the various structures described below can be appropriately combined to implement within the scope that does not cause technical contradictions. In addition, for components having the same functions as the components described in the above-mentioned embodiments, the same figure marks are marked and their descriptions are omitted. In addition, when only a part of the structure is mentioned, the structure of the embodiment described above can be applied to the other parts.

[0120] <Notice of support suspension>

[0121] The driving assistance ECU 20 may also be configured to implement a process of notifying the user that the signal light passing assistance may be stopped, i.e., a stop notice process, before deciding and notifying the stop of the signal light passing assistance. For example, the driving assistance ECU 20 may output a message indicating that the signal light passing assistance may be stopped when the remaining distance Rd is less than the stop notice distance Rn and the lighting device corresponding to the front signal light cannot be determined. The stop notice distance Rn is set to a range larger than the assistance stop distance Rth. The stop notice distance is equivalent to the second specified distance. For example, the stop notice distance Rn may be 70m or 100m. The stop notice distance Rn may be a value obtained by adding a specified margin such as 20m or 30m to the assistance stop distance Rth. The stop notice distance Rn and the margin may also be dynamically determined based on the driving speed of the vehicle acquired by the vehicle state acquisition unit F4. For example, the margin may also be dynamically determined to be a distance equivalent to 4 seconds. The message output as the stop notice process may also be an image message or a voice message. The display 151 may display a predetermined icon image indicating that the traffic light passing assistance may be stopped.

[0122] In such a structure, in the case of a final stop signal light passing assistance, such as Figure 6 As shown, when passing a point where the remaining distance Rd is equal to the stop notification distance Rn, the driving assistance ECU 20 notifies the user that the traffic light passing assistance may be stopped. Furthermore, when passing a point where the remaining distance Rd is equal to the assistance stop distance Rth, the driving assistance ECU 20 notifies the user that the traffic light passing assistance is stopped. According to such a structure, the user can know the possibility of stopping the traffic light passing assistance in advance, in other words, at a stage where there is enough time. As a result, when the traffic light passing assistance is finally stopped, the possibility of causing panic to the user can be reduced.

[0123] <Supplementary information on the deemed threshold value Dth>

[0124] The signal light recognition unit F71 may also be configured to moderate the threshold Dth according to the signal light probability P of the detected lighting device. That is, the threshold Dth is set to a larger value as the signal light probability P of the detected lighting device increases. Figure 7 As shown, when the signal light probability P is 40%, the threshold value Dth is set to the predetermined first distance D1. On the other hand, when the signal light probability P is 90% or more, the threshold value Dth is set to the predetermined second distance D2. When the signal light probability P is less than 90%, the threshold value Dth is set to be larger as the signal light probability P is larger. The first distance D1 can be, for example, 0.5m, 0.7m, etc. In addition, the second distance D2 can be, for example, 2.0m, 2.5m, etc.

[0125] According to such a structure, the possibility of mistakenly determining that a traffic light is not a traffic light due to the fact that the setting position of the traffic light registered on the map, that is, the position of the traffic light on the map, is offset from the actual setting position can be reduced. That is, it is possible to prevent the situation where the traffic light cannot be detected when the position of the traffic light on the map is wrong. In addition, according to this structure, the degree of deviation between the position of the traffic light on the map and the position of the traffic light observed by the front camera 11, that is, the observed traffic light position, can be evaluated with high accuracy. In addition, as mentioned above, the setting position of the traffic light registered on the PD map includes errors. In view of such a situation, the above structure is particularly preferred when the map data processed by the map server 3 is a PD map.

[0126] <Supplementary information of the signal light recognition unit F71>

[0127] The above-mentioned signal light recognition method is not only applicable to the recognition of signal lights for vehicles, but can also be similarly applied to the recognition of signal lights for pedestrians. The signal light recognition unit F71 can also be configured to detect signal lights for pedestrians. If the signal lights for pedestrians can be detected, it is easy to predict the behavior of pedestrians based on the lighting state. That is, by using the lighting state of the signal lights for pedestrians to plan vehicle control when passing through intersections, etc., safety can be further improved.

[0128] In addition, the configuration may be such that, when it is determined by referring to the map data that both the vehicle signal light and the pedestrian signal light are provided for one direction of travel, the vehicle signal light and the pedestrian signal light are identified as a group. The configuration may also be such that, if the proximity of the vehicle signal light is less than a predetermined threshold and the proximity of the pedestrian signal light is also less than a predetermined threshold, it is determined to be a signal light. According to this configuration, since the judgment material increases, the possibility of erroneously determining that a structure that is not a signal light is a signal light can be reduced.

[0129] In addition, the signal light recognition unit F71 may also use the recognition result of the signal light for pedestrians as an auxiliary to the recognition of the signal light for vehicles. For example, when the signal light for pedestrians can be recognized, the light device that emits light of a color corresponding to the lighting state of the signal light for pedestrians may be preferentially searched. For example, when the signal light for pedestrians is green, the signal light may be searched from green light devices. Similarly, the detection result of the signal light for vehicles may be used as an auxiliary to the recognition of the signal light for pedestrians. The recognition result of the signal light for vehicles and the recognition result of the signal light for pedestrians can be used in a complementary manner.

[0130] In addition, the signal lights for pedestrians are arranged at a relatively lower position than the signal lights for vehicles. Therefore, the outer shell of the signal lights for pedestrians is easier to detect by the headlights of the vehicle, street lights, etc. than the signal lights for vehicles. In addition, even in the daytime, the outer shell of the signal lights for pedestrians is less likely to blend into the background such as clouds than the signal lights for vehicles. Therefore, as long as the field of view of the front camera 11 is not blocked by other vehicles, buildings, etc., the signal lights for pedestrians are sometimes easier to detect than the signal lights for vehicles. If such a situation is taken into consideration, the recognition accuracy of the signal lights for vehicles can be improved by complementarily using the recognition result of the signal lights for pedestrians to determine whether the detected lighting device is a signal light for vehicles.

[0131] On the other hand, the signal lights for vehicles are arranged at a relatively higher position than the signal lights for pedestrians. Therefore, the signal lights for vehicles have the characteristic of being easier to detect from a distance than the signal lights for pedestrians. In addition, in most cases, the signal lights for pedestrians are linked with the signal lights for vehicles. Therefore, in the search for signal lights for pedestrians, the lighting status information of the signal lights for vehicles can also be used. For example, when the signal lights for vehicles are green, green light sources can also be preferentially searched as signal lights for pedestrians. According to the structure of using the lighting status information of the signal lights for vehicles in the search for signal lights for pedestrians, the processing load for detecting signal lights for pedestrians can be reduced or the detection accuracy can be improved. Preferably, the map data includes not only the setting positions of the signal lights for vehicles but also the setting positions of the signal lights for pedestrians. In addition, in the map data, as signal light data, it is preferred to include data indicating whether the signal lights for vehicles and pedestrians with the same travel direction are synchronized, in other words, whether it is a separate signal for pedestrians.

[0132] When the pedestrian signal light and the vehicle signal light in the same direction are linked, the signal light recognition unit F71 may also infer the lighting state of the vehicle signal light based on the lighting state of the pedestrian signal light. In addition, there is sometimes a time difference of several seconds to 10 seconds between the lighting state change timing of the pedestrian signal light and the vehicle signal light. The signal light recognition unit F71 may also predict the change of the lighting state of the vehicle signal light based on the change of the lighting state of the pedestrian signal light, and prepare and correct the control plan.

[0133] Furthermore, when the signal light recognition unit F71 determines the light device corresponding to the signal light ahead, it uses a prescribed tracking method to track the light device as the signal light ahead. As a method for tracking a light device, a variety of methods can be cited. For example, the signal light recognition unit F71 infers the position of the signal light at the current moment based on the observed position of the signal light at the previous moment and the moving speed of the vehicle, and regards the light device closest to the inferred position from among the light devices included in the current image frame as the same object. In addition, for example, it can also be configured to track a signal light whose proximity δ calculated at the previous moment and the newly calculated proximity δ are within a prescribed threshold value as the same signal light. In addition, the signal light recognition unit F71 can also use the similarity of feature quantities such as color histogram, size, brightness, etc. to track the light device equivalent to the signal light.

[0134] In addition, in a configuration that mainly uses the similarity between the estimated position based on the observation position at the previous moment and the observation position for tracking, the probability that the lighting device is a signal light can be increased based on the fact that the lighting color of the lighting device changes according to a predetermined pattern. The change pattern for increasing the probability that the detected lighting device is a signal light includes, for example, a pattern that changes from green to yellow or red, or a pattern that changes from red or yellow to green. The change pattern for increasing the probability that the detected lighting device is a signal light can be appropriately changed to correspond to the change pattern of the signal light in the area where the driving assistance ECU 20 is used.

[0135] <About the use of map updates>

[0136] According to the above structure, the offset and direction between the position of the traffic light on the map and the observed traffic light position can be calculated. Figure 8 As shown in FIG. 1 , the driving assistance ECU 20 includes a report processing unit F10 as a functional unit, and the report processing unit F10 sends a data set indicating the degree of deviation between the signal light position on the map and the observed signal light position as detection data to the map server 3. Figure 8 The illustration of the already described structures such as the environment recognition unit F7 is omitted.

[0137] The detection data includes, for example, signal light identification information and signal light position offset information. The signal light identification information is information indicating the signal light to be reported, and may be, for example, the identification number of the signal light registered in the map data. The signal light identification information may also be a data set indicating the name or node number of the intersection and the direction of travel of the vehicle. Preferably, the signal light position offset information is information indicating the offset amount and direction of the signal light position on the map and the observed signal light position, including the offset amounts of the latitude, longitude and altitude. The signal light position offset information is equivalent to data indirectly indicating the observed signal light position. Alternatively, or in parallel with the signal light position offset information, the detection data may include data directly indicating the observed signal light position, i.e., coordinate data of the observed signal light position. The detection data including the signal light position offset information and / or the coordinate data of the observed signal light position is equivalent to the observation result information. The observed signal light position used for generating the detection data may also be a position calculated at the time when the remaining distance Rd to the signal light is a specified distance such as 40 m, or a position calculated at the time when the signal light is finally detected.

[0138] The detection data may also include an error evaluation value ρ at the moment of calculating the reported signal light position or signal light position offset information. By including the error evaluation value ρ in the detection data, the map server 3 can evaluate the accuracy of the reported signal light position information. In addition, the detection data may also include driving track information, route information, and ground feature information. Driving track information is information representing the track on which the vehicle is traveling. For example, the driving track information is expressed as a point sequence of the position of the vehicle. Route information is information representing the track of the end and center line of the driving path. In addition, the end of the driving path, etc. can also be expressed by a group of coordinate points. In addition, the driving track information and route information can also be expressed by a cubic spline curve. In addition, the driving track information and route information can also be expressed by coefficient parameters of a regression equation representing the detection points. The regression equation is a function equivalent to a straight line or curve that approximately represents the distribution of multiple detection points, for example, calculated by the least squares method, etc.

[0139] When the deviation of the position of the traffic light above the prescribed threshold is detected, the report processing unit F10 sends the detection data. In addition, the report processing unit F10 may send the detection data periodically. In addition, whether to send the detection data may be controlled based on instructions from the map server 3. In addition, as an embodiment, the determination of whether the detected lighting device is a traffic light based on the proximity δ is performed on the condition that the error evaluation value ρ is less than the error threshold Pth. That is, the above structure is equivalent to the following structure, in which the detection data for updating the position information of the traffic light is uploaded to the map server 3 on the condition that the error evaluation value ρ is less than the error threshold Pth.

[0140] like Figure 8 As shown, the map server 3 for updating the map includes a server processor 31 and a memory 32, and is configured as a computer. The server processor 31 is configured using a processor such as a CPU. The memory 32 includes a RAM and a flash memory. In addition, the map server 3 has a non-volatile storage device for storing map data, namely a map database 33. The map database 33 is configured to enable the server processor 31 to write, read, delete, etc. data. The map server 3 is configured to be able to communicate with the vehicle Ma via a communication device and a wide area communication network not shown.

[0141] The map server 3 includes a report receiving unit G1, a map updating unit G2, and a distribution processing unit G3 as functional units implemented by executing a predetermined map updating program by the server processor 31. The report receiving unit G1 acquires detection data uploaded from a plurality of vehicles including the own vehicle via a communication device. The report receiving unit G1 provides the detection data acquired from each vehicle to the map updating unit G2.

[0142] The map updating unit G2 updates the map data stored in the map database 33 based on the detection data sent from each vehicle. For example, by comprehensively processing the signal light position offset information reported from multiple vehicles for the same signal light, the correction amount of the signal light position is determined and the map data is updated. For example, for each of the latitude, longitude and altitude, the variance of the signal light position offset amount observed in multiple vehicles is calculated, and when the variance is less than a predetermined threshold, the median value / average value is adopted as the correction amount of the signal light position. For items whose variance is greater than the predetermined threshold, for example, a verification flag is established. The verification flag is equivalent to a flag indicating that data is uncertain as map registration. In addition, the map updating unit G2 may calculate the variance of the signal light position observed in multiple vehicles, and when the variance is less than a predetermined threshold, the median value / average value is adopted as the coordinates of the signal light. For signal lights whose variance is greater than the predetermined threshold, for example, a verification flag may be established and the update is withheld. In addition, the evaluation of the variance in the comprehensive processing is not necessary and may be omitted. The map updating unit G2 may also update the installation position of the traffic light when receiving a predetermined number of reports from a plurality of vehicles indicating that the position of the traffic light is shifted in the same direction. In addition, the map updating unit G2 preferably performs comprehensive processing by giving priority to, for example, increasing the weight of, the traffic light position observed when the error evaluation value ρ is smaller than the traffic light position observed when the error evaluation value ρ is larger. According to such a configuration, the accuracy of the installation position of the traffic light recorded in the PD map can be further improved.

[0143] The map updating unit G2 updates the map data at a predetermined period, for example. The update period may be every day, every week, or every month. In addition, the update may be performed sequentially from the traffic lights storing a predetermined amount of detection data. The number of detection data required to update the position coordinates of a traffic light, i.e., the number of update required reports, may be, for example, 10. The number of update required reports may also be 8, 12, etc. It can be expected that the greater the number of update required reports, the higher the accuracy. By reducing the number of update required reports, real-time performance can be improved.

[0144] The distribution processing unit G3 is a structure that distributes map data based on requests from vehicles in block units such as map images. For example, the map acquisition unit F2 of the vehicle requests the map server 3 for map data related to the current position and the area scheduled to pass within a specified time. The distribution processing unit G3 distributes the map data based on the request. In addition, the distribution processing unit G3 can also be configured to distribute map data spontaneously. For example, in a system structure in which map data is stored in a vehicle and used, updated map data can also be distributed to each vehicle as differential map data. The position information of three-dimensional structures such as traffic lights is distributed in a corresponding relationship with road surface marking data such as dividing lines and guide strips in intersections. In addition, the position information of three-dimensional structures such as traffic lights can also be distributed in a corresponding relationship with each travel direction and each lane at the intersection. According to the above structure, the position of the traffic light can be updated at any time based on the report from the vehicle.

[0145] <About cooperation with roadside equipment, etc.>

[0146] You can also Fig. 9 As shown in FIG. 1 , the driving assistance ECU 20 includes a signal light information receiving unit F11 as a functional unit. The signal light information receiving unit F11 is a structure that obtains information about the signal light ahead, that is, signal light information, from the roadside device via the V2X vehicle-mounted device 14. The signal light information may include, for example, the distance to the signal light ahead, the current lighting state, the lighting cycle, the remaining time until the lighting state changes, and the like.

[0147] For example, when the signal light information receiving unit F11 is able to receive the signal light information, the signal light identification unit F71 uses the received signal light information to identify the signal light. For example, when the signal light information is used to notify that the current lighting state is green, a correspondence is established between the signal light registered on the map and the lighting device detected by the front camera 11 from the green light. In addition, the signal light information from the roadside device may not be stably and continuously received once it is received until the signal light is passed. Even if the signal light information can be obtained up to the point where the remaining distance Rd is 100m, there is a situation where it cannot be received afterwards due to communication failure or radio wave obstruction by large vehicles. Therefore, once the signal light information is received, it is preferred not to rely on the signal light information, but to continue tracking so that the lighting state can also be identified using the front camera 11. As a tracking method, a variety of methods can be used as described above.

[0148] In addition, in a configuration in which the similarity of feature quantities is used to track a lighting device corresponding to a signal light, there is a possibility that the timing of the change in the lighting state of the signal light is lost. This is because the color changes greatly with the change in the lighting state of the signal light. In a configuration in which the similarity of color is used to track a lighting device corresponding to a signal light, it is preferred to complementarily use the signal light information received by the signal light information receiving unit F11 to track the lighting device as a signal light. For example, it may be configured so that the weight of the color information used in tracking the lighting device is reduced near the time when the lighting state of the signal light shown in the signal light information changes. Specifically, it is also possible to switch from a first tracking method using the similarity of color to a second tracking method not using the similarity of color to implement tracking near the time when the lighting state of the signal light shown in the signal light information changes. The second tracking method may be, for example, a method of tracking based on the similarity between the estimated position based on the observation position at the previous moment and the observation position at the current moment. According to this configuration, the possibility of interrupting the tracking of the lighting device as a signal light due to the change in the lighting state can be reduced. In addition, color information is used when searching for lighting devices corresponding to traffic lights. On the other hand, after the correspondence is successfully established, the weight of color information in the tracking process is reduced, thereby improving the accuracy of traffic light detection and reducing the possibility of loss.

[0149] Of course, the signal light information is not only used to determine the lighting device corresponding to the signal light ahead, but can also be directly used for signal light passing assistance. For example, the control plan unit F8 can also determine whether to stop at the signal light or make a timetable for acceleration and deceleration based on the signal light information. When the signal light information indicates that the light has not changed to red for more than a specified time, the control plan unit F8 can also formulate a control plan for passing the signal light without stopping. In addition, when it is foreseen based on the signal light information that the light will change to red before the remaining distance Rd is less than a specified value, a control plan for stopping and decelerating can be formulated. In addition, the acquisition source of the signal light information based on wireless communication is not limited to the roadside machine. The above-mentioned signal light information can also be obtained from a center that manages and distributes the operating status of the signal light. Generally, the signal light information wirelessly transmitted from the infrastructure is more reliable than the image recognition result. Therefore, when the signal light information can be received wirelessly, the control plan can be implemented based on the signal light information obtained from the infrastructure compared to the lighting state of the signal light determined by image recognition. However, there is a concern that a malicious third party will distribute forged signal light information. Based on such concerns, the control plan may be implemented by using the lighting state of the traffic light determined by image recognition in priority to the traffic light information obtained from the infrastructure. In addition, the reliability of the received traffic light information may be evaluated based on the matching between the traffic light information and the image recognition result, and then a control plan may be prepared based on the traffic light information.

[0150] <Modification of system configuration>

[0151] In the above-mentioned embodiment, the driving assistance ECU 20 is provided with a structure of a signal light recognition unit F71, but the arrangement of various functional units is not limited to this. Fig.10 As shown, the camera ECU 41 includes a signal light recognition unit F71. In addition, the camera ECU 41 and the driving assistance ECU 20 may be integrated as described above. In other words, the driving assistance ECU 20 may also have the function of the camera ECU 41. That is, the front camera 11 may also be configured to output image data to the driving assistance ECU 20, and the driving assistance ECU 20 may perform processing from object detection to type recognition.

[0152] In addition, it can also be configured to input the detection results of various surrounding monitoring sensors such as millimeter-wave radar, sonar, LiDAR, etc. into the driving assistance ECU20. For example, the environment recognition unit F7 can also be configured to recognize the driving environment through sensor fusion processing that integrates the detection results of millimeter-wave radar, LiDAR and the detection results of the front camera 11 with prescribed weights. In addition, the above illustrates the process of using the image of the front camera 11 to recognize the traffic light, but the camera used to recognize the traffic light is not limited to the front camera 11. It can also be configured to detect the traffic light using image frames generated by the rear camera, the right side camera, and the left side camera. The above-mentioned traffic light recognition method can also be applied to image data captured by vehicle-mounted cameras other than the front camera 11, such as side cameras.

[0153] <Additional Notes>

[0154] The control unit and processing unit and the method thereof described in the present invention may also be implemented by a special-purpose computer, which is a processor programmed to execute one or more functions embodied by a computer program. In addition, the device and the method thereof described in the present invention may also be implemented using a special-purpose hardware logic circuit. Furthermore, the device and the method thereof described in the present invention may also be implemented by one or more special-purpose computers, which are composed of a combination of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may also be stored as an instruction executed by a computer in a non-migratable tangible recording medium that can be read by a computer.

[0155] That is, the units and / or functions provided by the processing unit 21 and the like can be provided by software recorded in a physical memory device and a computer executing the software, software only, hardware only, or a combination thereof. For example, part or all of the functions of the processing unit 21 can also be implemented as hardware. In the method of implementing a certain function as hardware, there is a method of implementing it using one or more ICs, etc. The processing unit 21 can also replace the CPU and be implemented using an MPU, a GPU, or a DFP (Data Flow Processor). The processing unit 21 can also be implemented by combining multiple computing processing devices such as a CPU, an MPU, and a GPU. The processing unit 21 can also be implemented as a system-on-chip (SoC). In addition, various processing units can also be implemented using FPGA (Field-Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). Various programs can be stored in a non-transitory tangible storage medium. As a storage medium for the program, various storage media such as HDD (Hard-disk Drive), SSD (Solid State Drive), EPROM (Erasable Programmable ROM), flash memory, SD (Secure Digital) card, etc. can be used.

Claims

1. A signal light recognition device for recognizing signal lights existing around a vehicle, wherein: The signal light recognition device comprises: A vehicle position acquisition unit, which acquires the position of the vehicle; a position acquisition unit on a map, which acquires the position of the traffic light registered in the map data, that is, the position of the traffic light on the map, based on the position of the vehicle acquired by the vehicle position acquisition unit; an observation position acquisition unit, which acquires position information of the lighting device detected using an image frame captured by the vehicle-mounted camera as an observation lighting position; as well as a signal light recognition unit that determines whether the light device is the signal light based on a proximity, wherein the proximity is a difference between the signal light position on the map acquired by the position acquisition unit on the map and the observed light position acquired by the observed position acquisition unit, The traffic light recognition device is configured to send an error signal indicating a detection failure of the traffic light registered in the map to the map server when the lighting device corresponding to the traffic light cannot be determined even if the remaining distance to the traffic light registered in the map data is less than a specified distance.

2. The signal light recognition device according to claim 1, wherein: The error signal includes at least any one of current position information, observation time information, identification information of the signal light that failed to be detected, and model information of the vehicle-mounted camera.

3. The signal light recognition device according to claim 1, wherein: The signal light recognition unit is configured to determine that the lighting device is the signal light based on the fact that the proximity is smaller than a predetermined identification threshold.

4. The signal light recognition device according to claim 1, wherein: The signal light recognition device includes a position error acquisition unit that acquires an error evaluation value indicating a magnitude of an error in the position of the vehicle estimated by the vehicle position acquisition unit. The signal light recognition unit is configured to determine whether the lighting device is the signal light on the condition that the error evaluation value is smaller than a predetermined error threshold.

5. The signal light recognition device according to claim 4, wherein: The signal light recognition device is configured to send observation result information to a map server disposed outside the vehicle on the condition that the error evaluation value is less than the error threshold, wherein the observation result information directly or indirectly indicates an observation position of the lighting device corresponding to the signal light.

6. The signal light recognition device according to any one of claims 1 to 5, wherein: The vehicle position acquisition unit determines the position coordinates of the vehicle on the map based on the position information of a predetermined landmark detected using the image frame captured by the front camera and the position information of the landmark registered in the map data, The landmark includes at least one of guide signs, signal lights, poles, commercial signs and prescribed road surface markings set along the road. The vehicle position acquisition unit is configured not to use a traffic light as the landmark at least at night.

7. The signal light recognition device according to any one of claims 1 to 5, wherein: The signal light recognition device is configured to notify a user of information on a recognition state of the signal light when the lighting device corresponding to the signal light cannot be identified even if the remaining distance to the signal light registered in the map data is less than a predetermined distance.

8. The signal light recognition device according to any one of claims 1 to 5, wherein: The signal light recognition device includes a signal light information receiving unit that receives signal light information from a communication device provided outside the vehicle, wherein the signal light information is information about the signal light. The signal light recognition unit is configured to, when the signal light information reception unit receives the signal light information, specify the lighting device corresponding to the signal light using the signal light information.

9. The signal light recognition device according to claim 8, wherein: The signal light recognition unit is configured to, when the lighting device corresponding to the signal light is identified, track the lighting device as the signal light. The signal light recognition device is configured to change a method of tracking the lighting device as the signal light over time based on the signal light information received by the signal light information receiving unit.

10. A signal light recognition method, which is executed by at least one processor and is used to recognize signal lights existing around a vehicle, wherein: The signal light recognition method comprises: A vehicle position acquisition step, acquiring the position of the vehicle; A map position acquisition step, based on the position of the vehicle acquired in the vehicle position acquisition step, acquiring the position of the traffic light registered in the map data, that is, the position of the traffic light on the map; an observation position acquisition step, acquiring position information of the lighting device detected using an image frame captured by the vehicle-mounted camera as an observation lighting position; a signal light recognition step of determining whether the light device is the signal light based on a proximity, wherein the proximity is a difference between the signal light position on the map acquired in the map position acquisition step and the observed light position acquired in the observed position acquisition step; and The step of transmitting an error signal indicating that detection of the traffic light registered in the map has failed to the map server when the lighting device corresponding to the traffic light cannot be identified even if the remaining distance to the traffic light registered in the map data is less than a predetermined distance.

11. A vehicle control device for implementing vehicle control corresponding to the recognition state of a traffic light existing around the vehicle, wherein: The vehicle control device comprises: A vehicle position acquisition unit, which acquires the position of the vehicle; a position acquisition unit on a map, which acquires the position of the traffic light registered in the map data, that is, the position of the traffic light on the map, based on the position of the vehicle acquired by the vehicle position acquisition unit; an observation position acquisition unit, which acquires position information of the lighting device detected using an image frame captured by the vehicle-mounted camera as an observation lighting position; as well as a signal light recognition unit that determines whether the light device is the signal light based on a proximity, wherein the proximity is a difference between the signal light position on the map acquired by the position acquisition unit on the map and the observed light position acquired by the observed position acquisition unit, The vehicle control device is configured as follows: if the traffic light cannot be recognized even if the remaining distance from the vehicle to the traffic light position on the map is less than the first predetermined distance, notifying the user that the vehicle control is not performed, and When the traffic light cannot be recognized even if the remaining distance from the vehicle to the traffic light position on the map is a second predetermined distance greater than the first predetermined distance, the user is informed that the vehicle control may not be performed.

Images