Driving assistance device, vehicle, driving assistance method, and storage medium

Abstract

The present application relates to a driving assistance device, a vehicle, a driving assistance method, and a storage medium. The present application provides a technology capable of appropriately determining a signal light indicating whether or not the host vehicle can travel, thereby improving the safety of the vehicle. A driving assistance device that assists driving of a vehicle includes: a photographing mechanism that photographs a front of the vehicle; a determination mechanism that determines a signal light within an image obtained by the photographing mechanism; a detection mechanism that detects a setting height of the signal light determined by the determination mechanism based on the image; and a judgment mechanism that judges whether or not the signal light determined by the determination mechanism is an object signal light indicating whether or not the vehicle can travel based on the setting height detected by the detection mechanism.

Description

Technical Field

[0001] This invention relates to driving assistance devices, vehicles, driving assistance methods, and storage media. Background Technology

[0002] Patent Document 1 describes a technique in which, when one or more traffic lights are identified in an image obtained by an imaging device, the travel trajectory of the vehicle is inferred, and a traffic light set as a control input is determined from the one or more traffic lights based on the lateral position of each traffic light relative to the travel trajectory (travel lateral position) and the lateral position of each traffic light relative to the straight line in front of the vehicle (forward lateral position).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent No. 5883833 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] If, as described in Patent Document 1, the signal lights are determined as control inputs solely based on their lateral positions, pedestrian signal lights, flashing signal lights, and other signal lights that meet the lateral position requirements of signal lights but are not closely related to the vehicle may sometimes be mistakenly determined as control inputs (i.e., signal lights indicating whether the vehicle can proceed).

[0008] Therefore, the object of the present invention is to provide a technique for appropriately determining whether a vehicle is permitted to proceed using a signal light.

[0009] means for solving problems

[0010] To achieve the above objectives, a driving assistance device as an aspect of the present invention is a driving assistance device for assisting in driving a vehicle, characterized in that the driving assistance device comprises: a shooting mechanism for shooting a picture of the front of the vehicle; a determining mechanism for determining a traffic light in an image obtained by the shooting mechanism; a detection mechanism for detecting, based on the image, the setting height of the traffic light determined by the determining mechanism; and a judging mechanism for determining, based on the setting height detected by the detection mechanism, whether the traffic light determined by the determining mechanism is an object traffic light indicating whether the vehicle can proceed.

[0011] To achieve the above objectives, a driving assistance method according to one aspect of the present invention is a driving assistance method for assisting the driving of a vehicle, characterized in that the driving assistance method includes: a shooting step, in which a shot is taken of the front of the vehicle; a determination step, in which a signal light within the image obtained by the shooting step is determined; a detection step, in which the setting height of the signal light determined by the determination step is detected from the image; and a judgment step, in which, based on the setting height detected by the detection step, it is determined whether the signal light determined by the determination step is an object signal light indicating whether the vehicle can proceed.

[0012] Invention Effects

[0013] According to the present invention, for example, it is possible to provide a technology that can appropriately determine whether a vehicle is permitted to proceed, thereby improving vehicle safety. Attached Figure Description

[0014] Figure 1 It is a block diagram of the vehicle and its control devices.

[0015] Figure 2 This is a block diagram illustrating a structural example of a driver assistance device.

[0016] Figure 3 This is a diagram showing an example of a frontal image obtained through the camera unit.

[0017] Figure 4 This is a flowchart representing driver assistance processing.

[0018] Figure 5 This is a flowchart illustrating the process of determining whether a signal light is an object.

[0019] Figure 6 It is a map showing the regional differences in the location, height, lateral distance, and distance from the stop line of vehicle traffic lights.

[0020] Figure 7 This is a flowchart illustrating the process of determining whether an alarm is needed.

[0021] Figure 8 It is a diagram that represents a combination of information about the lighting status.

[0022] Explanation of reference numerals in the attached figures

[0023] 100: Driving assistance devices;

[0024] 110: Filming Department;

[0025] 120: Position Detection Department;

[0026] 130: Alarm Output Unit;

[0027] 140: Processing Department;

[0028] 141: Acquisition Department;

[0029] 142: Determining the department;

[0030] 143: Testing Department;

[0031] 144: Judgment section;

[0032] 145: Alarm Control Department. Detailed Implementation

[0033] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not intended to limit the invention to which the technical solution pertains. Additionally, not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more features from the plurality of features described in the embodiments may be arbitrarily combined. Furthermore, identical or identical components are labeled with the same reference numerals, and repeated descriptions are omitted.

[0034] One embodiment of the present invention will be described. Figure 1 This is a block diagram of the vehicle V and its control device 1 according to this embodiment. Figure 1 In this embodiment, the outline of vehicle V is shown in a top view and a side view. As an example, vehicle V in this embodiment can be a four-wheeled passenger car, such as a parallel hybrid vehicle. In this case, the power unit 50, which outputs the driving force to rotate the drive wheels of vehicle V, can include an internal combustion engine, a motor, and an automatic transmission. The motor can be used as a drive source to accelerate vehicle V, and can also be used as a generator during deceleration (regenerative braking). Furthermore, vehicle V is not limited to a four-wheeled passenger car; it can be a straddle-type vehicle (motorized two-wheeler, motorized three-wheeler), or a large vehicle such as a truck or bus.

[0035] [Structure of the vehicle's control system]

[0036] Reference Figure 1The structure of the control device 1, which is an on-board device of a vehicle V, will be described. The control device 1 can include an information processing unit 2 composed of multiple ECUs (Electronic Control Units) 20-28 capable of communicating with each other. Each ECU includes a processor, such as a CPU (Central Processing Unit), a storage device such as a semiconductor memory, and an interface for external devices. The storage device stores programs executed by the processor, data used by the processor for processing, etc. Each ECU may also have multiple processors, storage devices, and interfaces. Furthermore, the number and functions of the ECUs can be appropriately designed, and they can be subdivided or combined compared to this embodiment. For example, ECUs 20-28 can be composed of a single ECU. Furthermore, in Figure 1 The names of representative functions of ECUs 20 to 28 are marked in the text. For example, ECU 20 is marked as "Driving Control ECU".

[0037] ECU 20 performs controls related to the driving control of vehicle V, including driving assistance for vehicle V. In this embodiment, ECU 20 controls the driving (acceleration of vehicle V based on power unit 50, etc.), steering, and braking of vehicle V. Furthermore, in manual driving, ECU 20 can, for example, issue a warning to the driver or provide braking assistance to vehicle V when the indicator light for the vehicle V's movement is red (red light) or yellow (yellow light). This warning can be issued by displaying information on the display device of information output device 43A (described later), or by reporting information through sound or vibration. Additionally, the braking assistance can be performed by controlling the braking device 51.

[0038] ECU21 is an environment recognition unit that identifies the driving environment of vehicle V based on the detection results of detection units 31A, 31B, 32A, and 32B that detect the surrounding conditions of vehicle V. In this embodiment, ECU21 can detect the position of objects (e.g., obstacles, other vehicles) around vehicle V based on the detection results of at least one of detection units 31A, 31B, 32A, and 32B.

[0039] Detection units 31A, 31B, 32A, and 32B are sensors capable of detecting objects around the vehicle V (this vehicle). Detection units 31A and 31B are cameras (hereinafter sometimes referred to as cameras 31A and 31B) that capture images of the front of the vehicle V, mounted on the interior side of the front windshield of the vehicle V. By analyzing the images captured by cameras 31A and 31B, the outlines of objects and lane markings (white lines, etc.) on the road can be extracted. Furthermore, in this embodiment, two cameras 31A to 31B are provided on the vehicle V, but a structure with only one camera is also possible.

[0040] The detection unit 32A is a LiDAR (Light Detection and Ranging) (hereinafter sometimes referred to as LiDAR 32A), which detects objects around the vehicle V and detects (measures) the distance to the object and the direction (azimuth) of the object. Figure 1 In the example shown, five optical radars 32A are provided: one at each corner of the front of the vehicle V, one at the center of the rear, and one on each side of the rear. Alternatively, the optical radars 32A may not be provided in the vehicle V. Furthermore, the detection unit 32B is a millimeter-wave radar (hereinafter sometimes referred to as radar 32B), which uses radio waves to detect objects around the vehicle V, detecting (measuring) the distance to the object and the object's direction (azimuth). Figure 1 In the example shown, there are five radars 32B: one at the center of the front of the vehicle V, one at each of the front corners, and one at each of the rear corners.

[0041] ECU22 is a steering control unit that controls the electric power steering system 41. The electric power steering system 41 includes a mechanism that steers the front wheels according to the driver's driving operation (steering operation) on the steering wheel ST. The electric power steering system 41 includes the following components: a drive unit 41a that provides a driving force (sometimes referred to as steering assist torque) for assisting steering operation or for automatically steering the front wheels; a steering angle sensor 41b; and a torque sensor 41c that detects the steering torque borne by the driver (referred to as steering load torque, to be distinguished from steering assist torque).

[0042] ECU 23 is a brake control unit that controls the hydraulic system 42. The driver's braking operation on the brake pedal BP is converted into hydraulic pressure in the master cylinder BM and transmitted to the hydraulic system 42. The hydraulic system 42 is an actuator that controls the hydraulic fluid supplied to the brake systems (e.g., disc brakes) 51 located on each of the four wheels based on the hydraulic pressure transmitted from the master cylinder BM. ECU 23 performs drive control on the solenoid valves and other components included in the hydraulic system 42. Furthermore, ECU 23 can illuminate the brake lights 43B during braking. This increases the driver's attention to the vehicle V relative to following vehicles.

[0043] The ECU 23 and the hydraulic device 42 can constitute an electric servo brake. The ECU 23 can, for example, control the distribution of braking force based on the four braking devices 51 and the regenerative braking force based on the motor of the power unit 50. The ECU 23 can also implement ABS, traction control, and vehicle V attitude control functions based on the detection results of wheel speed sensors 38, yaw rate sensors (not shown), and pressure sensors 35 that detect pressure in the master cylinder BM, which are respectively located on the four wheels.

[0044] ECU24 is a stop-and-hold control unit that controls the electric parking brake 52 installed on the rear wheels. The electric parking brake 52 has a mechanism for locking the rear wheels. ECU24 can control the locking and unlocking of the rear wheels based on the electric parking brake 52.

[0045] ECU 25 is an in-vehicle reporting control unit that controls the information output device 43A for reporting information to the vehicle interior. The information output device 43A may include, for example, a display device installed on a head-up display, an instrument panel, or an audio output device. Furthermore, it may also include a vibration device. ECU 25, for example, causes the information output device 43A to output various information such as vehicle speed, outside temperature, route guidance, and information related to the state of the vehicle (V).

[0046] ECU 26 is equipped with a communication device 26a for wireless communication. The communication device 26a can exchange information wirelessly with objects that have communication capabilities. Examples of objects with communication capabilities include vehicles (vehicle-to-vehicle communication), fixed equipment such as traffic lights and traffic monitoring devices (road-to-road communication), and people carrying portable terminals such as smartphones (pedestrians, motor vehicles). Furthermore, ECU 26 can access various information, such as road information, by accessing servers on the Internet through the communication device 26a.

[0047] ECU 27 is a drive control unit that controls the power unit 50. In this embodiment, one ECU 27 is assigned to the power unit 50, but it is also possible to assign one ECU to each of the internal combustion engine, the motor, and the automatic transmission. ECU 27 controls the output of the internal combustion engine and the motor, or switches the gears of the automatic transmission, in accordance with the driver's driving operations and vehicle speed detected by the operation detection sensor 34a provided on the accelerator pedal AP and the operation detection sensor 34b provided on the brake pedal BP. Furthermore, in the automatic transmission, a speed sensor 39 is provided as a sensor for detecting the driving state of the vehicle V, which detects the rotational speed of the output shaft of the automatic transmission. The vehicle speed of the vehicle V can be calculated from the detection result of the speed sensor 39.

[0048] ECU 28 is a position recognition unit that identifies the current position and travel route of vehicle V. ECU 28 controls the gyroscope sensor 33, GPS (Global Positioning System) sensor 28b, and communication device 28c, and processes the detection or communication results. The gyroscope sensor 33 detects the rotational motion (yaw rate) of vehicle V. The travel route of vehicle V can be determined based on the detection results of the gyroscope sensor 33, etc. The GPS sensor 28b detects the current position of vehicle V. The communication device 28c wirelessly communicates with a server providing map information and traffic information to obtain this information. High-precision map information can be stored in database 28a, and ECU 28 can determine the position of vehicle V on the lane based on this map information, etc. In addition, vehicle V may also be equipped with a speed sensor to detect the speed of vehicle V, an acceleration sensor to detect the acceleration of vehicle V, and a lateral acceleration sensor (lateral G-sensor) to detect the lateral acceleration of vehicle V.

[0049] [Structure of driver assistance devices]

[0050] Figure 2 This is a block diagram illustrating a structural example of the driving assistance device 100 according to this embodiment. The driving assistance device 100 is a device for assisting a driver in driving a vehicle V, and for example, it may include a camera unit 110, a position detection unit 120, an alarm output unit 130, and a processing unit 140. The camera unit 110, the position detection unit 120, the alarm output unit 130, and the processing unit 140 are communicatively connected to each other via a system bus.

[0051] For example, the shooting unit 110 is Figure 1 The cameras 31A-31B shown photograph the front of the vehicle V. The position detection unit 120 is, for example,... Figure 1 The GPS sensor 28b shown detects the current position and direction of travel of the vehicle V. In addition to the GPS sensor 28b, the position detection unit 120 may also include a gyroscope sensor 33. Furthermore, the alarm output unit 130 is, for example... Figure 1 The information output device 43A shown reports various information to vehicle occupants (e.g., the driver) through displays on a monitor, sound outputs, etc. In this embodiment, the alarm output unit 130 can output an alarm to notify the driver of the illumination status when the indicator light indicating whether the vehicle V can proceed is red (red light) or yellow (yellow light).

[0052] The processing unit 140 comprises a computer including a processor (CPU, represented by a CPU), storage devices such as semiconductor memory, and interfaces for connecting to external devices, and is capable of functioning as... Figure 1 A portion of the ECU in the information processing unit 2 shown functions. A program (driving assistance program) for driving assistance of vehicle V is stored in the storage device, and the processing unit 140 can read and execute the driving assistance program stored in the storage device. In this embodiment, the processing unit 140 may include an acquisition unit 141, a determination unit 142, a detection unit 143, a judgment unit 144, and an alarm control unit 145.

[0053] The acquisition unit 141 acquires various information from sensors installed on the vehicle. In this embodiment, the acquisition unit 141 acquires an image obtained by the imaging unit 110 and the position information (current position information) of the vehicle V obtained by the position detection unit 120. The determination unit 142 performs image processing on the image obtained by the imaging unit 110 to determine the traffic lights included in the image. The detection unit 143 performs image processing on the image obtained by the imaging unit 110 to detect (calculate) the installation height of the traffic lights determined by the determination unit 142 based on the image. In this embodiment, the installation height of the traffic lights can be defined as the height of the traffic lights relative to the road surface where the traffic lights are installed, that is, the height from the road surface (the base of the traffic light support) at the location where the traffic lights are installed to the traffic lights.

[0054] The determination unit 144 determines, based on the setting height detected by the detection unit 143, whether the signal light determined by the determination unit 142 is a signal light indicating whether the vehicle V can proceed relative to the road where the vehicle V is traveling (hereinafter, sometimes referred to as an object signal light). If the determination unit 144 determines that the signal light determined by the determination unit 142 is an object signal light, the alarm control unit 145 determines, based on the illumination status of the object signal light, whether an alarm needs to be issued to the driver of the vehicle V. Then, if an alarm is determined to be required, the alarm output unit 130 is controlled to output an alarm to the driver of the vehicle V.

[0055] However, in the images obtained by the imaging unit 110, in addition to the traffic lights (object traffic lights) indicating whether the vehicle V can proceed, there may sometimes be traffic lights for intersections that are set up relative to the road where the vehicle V is traveling, or pedestrian traffic lights, flashing traffic lights, etc. Figure 3 An example of an image obtained by the imaging unit 110 (front image 60). Figure 3The forward image 60 shown is an image captured by the camera unit 110 when vehicle V approaches an intersection. In this forward image 60, in addition to the target traffic light 61, there are intersection traffic lights 62, pedestrian traffic lights 63, and flashing lights 64. Furthermore, the forward image 60 includes a stop line 65 where vehicle V should stop. Since the intersection traffic lights 62, pedestrian traffic lights 63, and flashing lights 64 have similar structures to the target traffic light 61, they may be misidentified as the target traffic light 61. Therefore, there is a need for technology to appropriately distinguish and identify the target traffic light 61 relative to the intersection traffic lights 62, pedestrian traffic lights 63, and flashing lights 64. In particular, there is a need for technology to appropriately distinguish and identify the pedestrian traffic lights 63 and flashing lights 64 from the target traffic light 61.

[0056] Therefore, as described above, the driving assistance device 100 (processing unit 140) of this embodiment includes: a detection unit 143 that detects the installation height of the traffic light determined by the determination unit 142; and a determination unit 144 that determines whether the traffic light determined by the determination unit 142 is a target traffic light based on the installation height detected by the detection unit 143. Since the pedestrian traffic light 63 and the flashing traffic light 64 are installed at a lower height compared to the vehicle traffic lights, the driving assistance device 100 of this embodiment can appropriately distinguish and identify the target traffic light 61 relative to the pedestrian traffic light 63 and the flashing traffic light 64.

[0057] [Driver Assistance Processing]

[0058] The driving assistance processing of this embodiment will be described below. Figure 4 This is a flowchart illustrating the driving assistance processing of this embodiment. Figure 4 The driving assistance processing shown in the flowchart is the processing performed by the processing unit 140 when the driving assistance program is executed in the driving assistance device 100.

[0059] In step S101, the processing unit 140 (acquisition unit 141) acquires an image (front image) obtained by the imaging unit 110 capturing the front of the vehicle V. Next, in step S102, the processing unit 140 (determination unit 142) performs image processing on the front image obtained in step S101 to determine the traffic lights included in the front image. For example, the determination unit 142 can determine all traffic lights included in the front image by extracting portions that emit light in blue (green), yellow, or red. Here, known image processing methods can be used for the image processing performed by the determination unit 142. Furthermore, the traffic lights determined by the determination unit 142 include not only vehicle traffic lights but also pedestrian traffic lights and flashing lights. Figure 3 In the example, the determining unit 142 determines the vehicle traffic lights 61-62, the pedestrian traffic lights 63, and the flashing traffic lights 64 in the forward image 60.

[0060] In step S103, the processing unit 140 determines whether a traffic light was identified in the forward image in step S102. If no traffic light was identified in the forward image, the process proceeds to step S108; if a traffic light was identified in the forward image, the process proceeds to step S104. In step S104, the processing unit 140 (detection unit 143, determination unit 144) determines whether the traffic light identified in step S102 is a target traffic light indicating whether vehicle V (this vehicle) can proceed. The specific processing performed in step S104 will be described later. Next, in step S105, the processing unit 140 determines whether it was determined to be a target traffic light in step S104. If it was not determined to be a target traffic light, the process proceeds to step S108; if it was determined to be a target traffic light, the process proceeds to step S106.

[0061] In step S106, the processing unit 140 (determination unit 144, alarm control unit 145) determines whether an alarm needs to be issued to the driver based on the illumination status of the target traffic light. The specific processing performed in step S106 will be described later. If it is determined that an alarm is not needed, the process proceeds to step S108; if it is determined that an alarm is needed, the process proceeds to step S107. In step S107, the processing unit (alarm control unit 145) outputs an alarm to the driver by controlling the alarm output unit 130. Furthermore, while this embodiment shows an example of outputting an alarm to the driver, braking assistance may also be performed in addition to or in place of an alarm.

[0062] In step S108, the processing unit 140 determines whether to terminate the driving assistance of the vehicle V. For example, the processing unit 140 may determine to terminate the driving assistance of the vehicle V if the driver turns off the driving assistance of the vehicle V or if the ignition switch of the vehicle V is turned off. If the driving assistance of the vehicle V is not terminated, the process returns to step S101.

[0063] [Determination and processing of whether it is an object traffic light (S104)]

[0064] Next, referring to Figure 5 For Figure 4 The specific processing content of "determining whether it is an object signal light" in step S104 will be explained. Figure 5 It means in Figure 4 The flowchart shows the processing performed by the processing unit 140 (detection unit 143, judgment unit 144) in step S104.

[0065] In step S201, the processing unit 140 (detection unit 143) detects (calculates) the installation height of the traffic light determined in step S102 based on the forward image. The installation height, as described above, is defined as the height of the traffic light relative to the road surface where the traffic light is installed. Figure 3 The height is recorded as "h". The detection unit 143 can detect the setting height of each signal light determined in step S102 by performing known image processing on the image in front.

[0066] Here, for example, there may be an angle (slope) between the road surface where vehicle V is located and the road surface where the traffic light is installed, or the road surface where the traffic light is installed (the base of the traffic light support) may not be included in the forward image. In this case, it may become difficult to accurately detect (calculate) the installation height of the traffic light based on the forward image. Therefore, the detection unit 143 can also calculate the height of the traffic light with vehicle V as a reference based on the forward image, and determine the installation height of the traffic light by modifying the vehicle reference traffic light height calculated based on the forward image based on the elevation difference information indicating the elevation difference between the road surface where vehicle V is located and the road surface where the traffic light is installed. The elevation difference information may be contained in map information stored in database 28a, and can be obtained from database 28a via acquisition unit 141. The detection unit 143 can obtain the elevation difference information from the map information obtained by acquisition unit 141 based on the current position of vehicle V detected by position detection unit 120 (GPS sensor 28b). Furthermore, elevation difference information can also be obtained from an external server via the acquisition unit 141 and the communication device 28c based on the current position of the vehicle V detected by the position detection unit 120.

[0067] In step S202, the processing unit 140 (determination unit 144) determines whether the setting height detected in step S201 meets a predetermined condition (height condition) related to the setting height of the vehicle signal light (target signal light). For example, the determination unit 144 can determine whether the height condition is met based on whether the setting height detected in step S201 is within a predetermined range. If the setting height does not meet the height condition, the process proceeds to step S210, determining that the signal light identified in step S102 is not a target signal light. On the other hand, if the setting height meets the height condition, the process proceeds to step S203. Through this step S202, it is possible to appropriately distinguish and identify whether the signal light identified in step S102 is a vehicle signal light, a pedestrian signal light, or a flashing signal light.

[0068] Here, the height at which vehicle signal lights are set varies by region (e.g., by country). Figure 6 This indicates regional differences in the location, height, lateral distance, and distance from the stop line of vehicle traffic lights. Figure 6 The example illustrates regions A through D, showing that the installation height of vehicle traffic lights varies depending on the region. Therefore, the determination unit 144 can also change the height conditions (i.e., the range of installation heights for determining target traffic lights) based on the region where vehicle V is traveling. Specifically, the determination unit 144 determines the region (e.g., country) where vehicle V is traveling based on the current position of vehicle V detected by the position detection unit 120, and changes the height conditions accordingly. Information indicating the height conditions based on the region can be stored, for example, in the database 28a, the memory of the processing unit 140, or obtained from an external server via the acquisition unit 141 and the communication device 28c.

[0069] In step S203, the processing unit 140 (detection unit 143) detects (calculates) the lateral distance between the traffic light and the vehicle V determined in step S102 based on the forward image. The lateral distance is defined as the lateral distance between a representative position (e.g., the center position) of the traffic light and a representative position (e.g., the center position) of the vehicle V. Figure 3 The distance is recorded as "L1". Lateral can be understood as the width direction of the vehicle V. The detection unit 143 can detect the lateral distance of each traffic light determined in step S102 by performing known image processing on the image in front.

[0070] In step S204, the processing unit 140 (determination unit 144) determines whether the lateral distance detected in step S203 meets a predetermined condition (first distance condition) related to the lateral distance of the target traffic light. For example, the determination unit 144 can determine whether the first distance condition is met based on whether the lateral distance detected in step S203 is within a predetermined range. If the lateral distance does not meet the first distance condition, the process proceeds to step S210, determining that the traffic light identified in step S102 is not the target traffic light. On the other hand, if the lateral distance meets the first distance condition, the process proceeds to step S205. Through this step S204, it is possible to appropriately distinguish and identify whether the traffic light identified in step S102 is a target traffic light indicating whether vehicle V can proceed or a crossroads traffic light.

[0071] Here, as Figure 6 As shown, the lateral distance of the target traffic light varies according to region (e.g., country). Therefore, the determination unit 144 can also change the first distance condition (i.e., the range used to determine the lateral distance of the target traffic light) according to the region where the vehicle V is traveling. Specifically, similar to the height condition, the determination unit 144 determines the region (e.g., country) where the vehicle V is traveling based on the current position of the vehicle V detected by the position detection unit 120, and changes the first distance condition according to the determined region. Information indicating the first distance condition according to region can be stored, for example, in the database 28a, the memory of the processing unit 140, or obtained from an external server via the acquisition unit 141 and the communication device 28c.

[0072] In step S205, the processing unit 140 (detection unit 143) detects (calculates) the distance between the traffic light and the vehicle V in the direction of travel, as determined in step S102, based on the forward image. The distance in the direction of travel is defined as the distance in the direction of travel between the representative position (e.g., the center position) of the traffic light and the representative position (e.g., the center position) of the vehicle V. Figure 3 The symbol is recorded as "L2". The direction of travel can be understood as the forward and backward direction of the vehicle V. The detection unit 143 can detect the distance of each traffic light in the direction of travel determined in step S102 by performing known image processing on the image in front.

[0073] In step S206, the processing unit 140 (determination unit 144) determines whether the travel direction distance detected in step S205 meets a predetermined condition (second distance condition) related to the travel direction distance of the target traffic light. For example, the determination unit 144 can determine whether the second distance condition is met based on whether the travel direction distance detected in step S205 is within a predetermined range. If the travel direction distance does not meet the second distance condition, the process proceeds to step S210, determining that the traffic light determined in step S102 is not the target traffic light. On the other hand, if the travel direction distance meets the second distance condition, the process proceeds to step S207. Through this step S206, it is possible to appropriately distinguish and identify whether the traffic light determined in step S102 is a traffic light located at the intersection where vehicle V is located or a traffic light located at an intersection preceding the intersection where vehicle V is located.

[0074] In step S207, the processing unit 140 (detection unit 143) detects the stop line set in the driving lane of the vehicle V based on the forward image, and detects the distance between the traffic light determined in step S102 and the traffic light (hereinafter, sometimes referred to as the stop line reference distance). The stop line reference distance can be defined as the distance in the direction of travel between the representative position (e.g., center position) of the traffic light and the representative position (e.g., center position) of the stop line. Figure 3 The image shows a stop line 65 set in the driving lane of vehicle V, and the reference distance of the stop line is recorded as "L3". The detection unit 143 can detect the stop line and the reference distance of the stop line of each traffic light determined in step S102 by performing known image processing on the image ahead.

[0075] In step S208, the processing unit 140 (determination unit 144) determines whether the stop line reference distance detected in step S207 meets a predetermined condition (third distance condition) related to the stop line reference distance of the target traffic light. For example, the determination unit 144 can determine whether the third distance condition is met based on whether the stop line reference distance detected in step S207 is within a predetermined range. If the stop line reference distance does not meet the third distance condition, the process proceeds to step S210, determining that the traffic light determined in step S102 is not the target traffic light. On the other hand, if the stop line reference distance meets the third distance condition, the process proceeds to step S209, determining that the traffic light determined in step S102 is the target traffic light. Through this step S208, it is possible to further appropriately distinguish and identify whether the traffic light determined in step S102 is the target traffic light indicating whether the vehicle V can proceed or a crossroads traffic light.

[0076] Here, as Figure 6As shown, the reference distance to the stop line of the target traffic light varies depending on the region (e.g., by country). Therefore, the determination unit 144 can also change the third distance condition (i.e., the range used to determine the stop line reference distance of the target traffic light) based on the region where the vehicle V is traveling. Specifically, similar to the height condition and the first distance condition, the determination unit 144 determines the region (e.g., by country) where the vehicle V is traveling based on the current position of the vehicle V detected by the position detection unit 120, and changes the third distance condition according to the determined region. Information indicating the third distance condition according to the region can be stored, for example, in the database 28a, the memory of the processing unit 140, or obtained from an external server via the acquisition unit 141 and the communication device 28c.

[0077] The above description illustrates an example of determining whether a traffic light is an object traffic light based on its setting height, lateral distance, travel direction distance, and stop line reference distance within the preceding image. However, this determination is not limited to the above; it can be based solely on the signal light's setting height, or it can be based on at least one of the lateral distance, travel direction distance, and stop line reference distance in addition to the setting height.

[0078] [Determination and processing of whether an alarm is needed (S106)]

[0079] Next, refer to Figure 7 right Figure 4 The specific processing content of "whether an alarm is needed" in step S106 will be explained. Figure 7 It means in Figure 4 A flowchart of the processing content performed by the processing unit 140 (judgment unit 144, alarm control unit 145) in step S106.

[0080] In step S301, the processing unit 140 (determination unit 144) determines whether there are multiple target traffic lights. In other words, if the alarm control unit 145 determines in step S102 that there are multiple traffic lights, then it determines whether there are multiple traffic lights that were identified as target traffic lights in step S104. If there are multiple target traffic lights, the process proceeds to step S302. On the other hand, if there are no multiple target traffic lights (i.e., if there is only one traffic light that was identified as a target traffic light in step S104), the process proceeds to step S304.

[0081] First, the case where multiple target indicator lights are identified in step S301 will be explained. In this case, steps S302 to S303 and S305 will be executed.

[0082] In step S302, the processing unit 140 (determination unit 144) sets a first candidate and a second candidate for the target traffic light from among the multiple traffic lights determined to be target traffic lights in step S104. For example, based on the detection result of the detection unit 143, the determination unit 144 sets (determines) the traffic light that meets the height condition and has the shortest lateral distance among the multiple traffic lights determined to be target traffic lights in step S104 as the first candidate for the target traffic light. Additionally, based on the detection result of the detection unit 143, the determination unit 144 sets (determines) the traffic light that meets the height condition and has the shortest distance in the traveling direction among the multiple traffic lights determined to be target traffic lights in step S104 as the second candidate for the target traffic light. Figure 3 In the example, since signal light 61 is the signal light whose set height h satisfies the height condition and whose lateral distance L1 is the shortest, it can be set as the first candidate for the target signal light. Furthermore, since signal light 62 is the signal light whose set height h satisfies the height condition and whose travel direction distance L2 is the shortest, it can be set as the second candidate for the target signal light. In addition, the "detection result of detection unit 143" used in step S302 is the result detected (calculated) in step S104, and includes at least the set height, lateral distance, and travel direction distance.

[0083] In step S303, the processing unit 140 (alarm control unit 145) detects the first candidate signal light ( Figure 3 In the example, traffic light 61) and the second candidate traffic light ( Figure 3 In the example, this is a combination of the lighting conditions of traffic light 62. For example, the alarm control unit 145 performs known image processing on the forward image acquired in step S101, detecting whether the lighting condition of each of the first candidate traffic light and the second candidate traffic light in the forward image is green (green light), yellow (yellow light), or red (red light). Thus, a combination of the lighting conditions of the first candidate traffic light and the second candidate traffic light can be obtained.

[0084] In step S305, the processing unit 140 (alarm control unit 145) determines whether the combination of lighting conditions detected in step S303 meets the stopping condition. The stopping condition refers to the condition under which vehicle V should stop at the intersection ahead of vehicle V. If the combination of lighting conditions meets the stopping condition, the process proceeds to step S306; otherwise, it proceeds to step S308.

[0085] For example, the alarm control unit 145 can be based on Figure 8 The combined information shown is used to determine whether the combination of lighting conditions detected in step S303 meets the stop condition. Figure 8The combined information shown is used to determine which of the first and second candidate traffic lights should be used as the target traffic light based on the combination of the illumination status of the first and second candidate traffic lights. For example, in the case where the first candidate traffic light is red and the illumination status of the second candidate traffic light is unknown (case [*1]), the first candidate traffic light is used as the target traffic light. Similarly, in the case where both the first and second candidate traffic lights are red (case [*2]), the first candidate traffic light is also used as the target traffic light. On the other hand, in the case where the illumination status of the first candidate traffic light is unknown and the second candidate traffic light is red (case [*3]), the second candidate traffic light is used as the target traffic light. These cases (cases [*1] to [*3]) are combinations of illumination status that satisfy the stop condition, and are cases where a warning to the driver is highly likely (warning target condition). That is, if the combination of illumination status detected in step S303 corresponds to any one of [*1] to [*3], the stop condition is satisfied.

[0086] Next, we will explain the case in step S301 where it is determined that there are no multiple object traffic lights (i.e., there is only one object traffic light). In this case, steps S304 to S305 are executed.

[0087] In step S304, the processing unit 140 (alarm control unit 145) detects the illumination status of the traffic light identified as an obstacle traffic light in step S104. For example, the alarm control unit 145 performs known image processing on the forward image acquired in step S101 to detect whether the illumination status of the obstacle traffic light in the forward image is green (green light), yellow (yellow light), or red (red light). Next, in step S305, the processing unit 140 (alarm control unit 145) determines whether the illumination status of the obstacle traffic light detected in step S304 meets the stop condition. For example, if the illumination status of the obstacle traffic light detected in step S304 is red or yellow, the alarm control unit 145 determines that the stop condition is met. If the illumination status of the obstacle traffic light meets the stop condition, the process proceeds to step S306; otherwise, it proceeds to step S308.

[0088] In step S306, the processing unit 140 (alarm control unit 145) acquires the speed of the vehicle V from the speed sensor via the acquisition unit 141 and determines whether the vehicle speed exceeds a threshold. If the vehicle speed exceeds the threshold, there is a higher probability that the driver will not notice the illumination status of the target indicator light (red or yellow). Therefore, in step S307, the alarm control unit 145 determines that an alarm for the driver is required, and then proceeds to... Figure 4 Step S107. On the other hand, if the vehicle speed does not exceed the threshold, the driver is more likely to notice the illuminated target signal light and want to stop the vehicle V. Therefore, in step S308, the alarm control unit 145 determines that an alarm for the driver is not required, and then proceeds to... Figure 4 Step S108. Furthermore, the vehicle speed threshold can be set arbitrarily, for example, it can be set to a speed that can be determined as the driver's intention to stop (for example, 5 to 20 km / h).

[0089] As described above, the driving assistance device 100 of this embodiment detects the setting height of a traffic light determined from the forward image obtained by the imaging unit 110, and determines whether the traffic light is a target traffic light indicating whether the vehicle V can proceed based on the setting height. Therefore, even when the forward image includes pedestrian traffic lights and flashing lights, the target traffic light can be appropriately distinguished and identified (determined) relative to these pedestrian traffic lights and flashing lights.

[0090] <Other Implementation Methods>

[0091] A program that performs one or more functions as described in the above embodiments is supplied to a system or device via a network or storage medium, and one or more processors in the computer of the system or device can read and execute the program. This is also how the present invention can be implemented.

[0092] <Summary of Implementation Methods>

[0093] 1. The driving assistance device described in the above embodiments is a driving assistance device (e.g., 100) that assists in driving a vehicle (e.g., V).

[0094] The driving assistance device includes:

[0095] A camera (e.g., 110) that films the front of the vehicle;

[0096] A determining mechanism (e.g., 142) determines the signal lights (e.g., 61-64) in the image (e.g., 60) obtained by the capturing mechanism;

[0097] A detection mechanism (e.g., 143) detects, based on the image, the installation height (e.g., h) of the traffic light determined by the determining mechanism; and

[0098] The determining mechanism (e.g., 144) determines, based on the setting height detected by the detection mechanism, whether the signal light determined by the determining mechanism is an object signal light indicating whether the vehicle can proceed.

[0099] According to this embodiment, even when the image obtained by the imaging mechanism includes pedestrian traffic lights and flashing lights, it is possible to appropriately distinguish and identify (determine) the object traffic lights indicating whether a vehicle can proceed relative to these pedestrian traffic lights and flashing lights.

[0100] 2. In the above embodiments,

[0101] If the setting height detected by the detection mechanism meets the predetermined conditions, the judgment mechanism determines that the signal light determined by the determination mechanism is the target signal light.

[0102] According to this embodiment, it is possible to appropriately identify the target signal light from the image obtained by the shooting mechanism.

[0103] 3. In the above embodiments,

[0104] The determination mechanism changes the predetermined conditions based on the region where the vehicle is traveling.

[0105] According to this embodiment, since predetermined conditions related to the installation height can be changed according to different regions, the target traffic light can be appropriately identified from the image obtained by the imaging mechanism based on the region.

[0106] 4. In the above embodiments,

[0107] The detection agency detects the height of the traffic light relative to the road surface where the traffic light is installed, and uses this height as the installation height.

[0108] According to this embodiment, since the setting height of each traffic light can be determined from the image obtained by the imaging mechanism using the same reference, the target traffic light can be appropriately identified from the image.

[0109] 5. In the above embodiments,

[0110] The detection agency calculates the height of the traffic light relative to the vehicle based on the image, and detects the installation height by modifying the height of the traffic light calculated based on the image, based on information indicating the height difference between the road surface where the vehicle is located and the road surface where the traffic light is installed.

[0111] According to this embodiment, even when there is an angle (slope) between the road surface where the vehicle is located and the road surface where the traffic lights are installed, or when the road surface where the traffic lights are installed (the base of the traffic light support) is not included in the image, the installation height of the traffic lights can be detected (calculated) with high accuracy.

[0112] 6. In the above embodiments,

[0113] The detection mechanism uses the image to detect the distance (e.g., L3) between the traffic light determined by the determining mechanism and the stop line (e.g., 65) set in the vehicle's driving lane.

[0114] The determining mechanism further determines whether the traffic light determined by the determining mechanism is the target traffic light based on the distance between the traffic light and the stop line determined by the determining mechanism.

[0115] According to this embodiment, it is possible to appropriately distinguish and identify whether a traffic light determined from an image is an object traffic light indicating whether a vehicle can proceed or a traffic light for crossroads.

[0116] 7. In the above embodiments,

[0117] The detection mechanism uses the image to detect the lateral distance (e.g., L1) between the traffic light and the vehicle, as determined by the determining mechanism.

[0118] The determining mechanism further determines whether the traffic light determined by the determining mechanism is the target traffic light based on the lateral distance detected by the detection mechanism.

[0119] According to this embodiment, it is possible to appropriately distinguish and identify whether a traffic light determined from an image is an object traffic light indicating whether a vehicle can proceed or a traffic light for crossroads.

[0120] 8. In the above embodiments,

[0121] The detection mechanism uses the image to detect the directional distance (e.g., L2) between the traffic light and the vehicle, as determined by the determining mechanism.

[0122] The determining mechanism further determines whether the traffic light determined by the determining mechanism is the target traffic light based on the travel direction distance detected by the detection mechanism.

[0123] According to this embodiment, it is possible to appropriately distinguish and identify whether the traffic light determined from the image is a traffic light located at the intersection where the vehicle is located or a traffic light located at an intersection preceding the intersection where the vehicle is located.

[0124] 9. In the above embodiments,

[0125] The driving assistance device also includes an alarm control mechanism (e.g., 130, 145), which outputs an alarm to the driver based on the illumination status of the target signal light when the judgment mechanism determines that the signal light determined by the determination mechanism is the target signal light.

[0126] According to this embodiment, vehicle safety can be improved because the driver can be appropriately notified of the illumination status of the target traffic light.

[0127] 10. In the above embodiments,

[0128] The alarm control mechanism determines that the alarm is output when the target signal light is lit in red or yellow and the vehicle's speed exceeds a threshold.

[0129] According to this embodiment, since the driver is more likely to not notice the illumination status (red or yellow) of the target signal light when the vehicle speed exceeds the threshold, the driver can be appropriately notified of the illumination status, thereby improving vehicle safety.

[0130] 11. In the above embodiments,

[0131] When multiple traffic lights are identified by the aforementioned determining mechanism

[0132] The detection mechanism, for each of the plurality of traffic lights, detects the set height (e.g., h), the lateral distance from the vehicle (e.g., L1), and the directional distance from the vehicle (e.g., L2) based on the image.

[0133] Based on the detection results of the detection mechanism, the judgment mechanism identifies the traffic light among the plurality of traffic lights that has the lowest setting height and the shortest lateral distance as the first candidate for the target traffic light, and identifies the traffic light that has the lowest setting height and the shortest distance in the direction of travel as the second candidate for the target traffic light.

[0134] The alarm control mechanism determines whether to output the alarm based on the combination of the lighting status of the first candidate signal light and the lighting status of the second candidate signal light.

[0135] According to this embodiment, when multiple traffic lights are identified based on an image, by setting multiple candidates related to the target traffic light and determining the presence or absence of an alarm output based on the combination of the illumination status of these multiple candidates, it is possible to accurately determine whether the vehicle can proceed and appropriately notify the driver of the alarm. In other words, vehicle safety can be improved.

[0136] This invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of this invention.

Claims

1. A driving assistance device, which assists in driving a vehicle, characterized in that, The driving assistance device includes: The camera is used to film the front of the vehicle. A determining mechanism that identifies the signal lights within the image obtained by the capturing mechanism; The detection agency, for the traffic light determined by the determining mechanism, detects the setting height, the lateral distance from the vehicle, and the travel direction distance from the vehicle based on the image; The judging mechanism determines, based on the detection results of the testing mechanism, whether the signal light determined by the determining mechanism is an object signal light indicating whether the vehicle can proceed. as well as The alarm control mechanism controls the output of alarms to the driver. When the determining mechanism identifies multiple traffic lights from the image, the detection mechanism detects the setting height, the lateral distance, and the travel direction distance for each of the multiple traffic lights based on the image. Based on the detection results of the detection mechanism, the judging mechanism identifies the traffic light among the multiple traffic lights whose setting height meets a predetermined condition and whose lateral distance is the shortest as the first candidate of the target traffic light, and identifies the traffic light whose setting height meets the predetermined condition and whose travel direction distance is the shortest as the second candidate of the target traffic light. The alarm control mechanism outputs the alarm based on the combination of the lighting status of the first candidate traffic light and the lighting status of the second candidate traffic light.

2. The driving assistance device according to claim 1, characterized in that, If the setting height detected by the detection mechanism meets the predetermined conditions, the judgment mechanism determines that the signal light determined by the determination mechanism is the target signal light.

3. The driving assistance device according to claim 1, characterized in that, The determination mechanism changes the predetermined conditions based on the region where the vehicle is traveling.

4. The driving assistance device according to claim 1, characterized in that, The detection agency detects the height of the traffic light relative to the road surface where the traffic light is installed, and uses this height as the installation height.

5. The driving assistance device according to claim 4, characterized in that, The detection agency calculates the height of the traffic light relative to the vehicle based on the image, and detects the installation height by modifying the height of the traffic light calculated based on the image, based on information indicating the height difference between the road surface where the vehicle is located and the road surface where the traffic light is installed.

6. The driving assistance device according to claim 1, characterized in that, The detection mechanism uses the image to detect the distance between the traffic light and the stop line set in the vehicle's driving lane, as determined by the determining mechanism. The determining mechanism further determines whether the traffic light determined by the determining mechanism is the target traffic light based on the distance between the traffic light and the stop line determined by the determining mechanism.

7. The driving assistance device according to claim 1, characterized in that, The detection mechanism uses the image to detect the lateral distance between the traffic light and the vehicle, as determined by the determining mechanism. The determining mechanism further determines whether the traffic light determined by the determining mechanism is the target traffic light based on the lateral distance detected by the detection mechanism.

8. The driving assistance device according to any one of claims 1 to 7, characterized in that, The detection mechanism uses the image to detect the directional distance between the traffic light and the vehicle, as determined by the determining mechanism. The determining mechanism further determines whether the traffic light determined by the determining mechanism is the target traffic light based on the travel direction distance detected by the detection mechanism.

9. The driving assistance device according to claim 1, characterized in that, When the determining mechanism identifies one target signal light from the image and the judging mechanism determines that the target signal light is present, the alarm control mechanism outputs an alarm based on the lighting status of the target signal light.

10. The driving assistance device according to claim 9, characterized in that, The alarm control mechanism outputs an alarm when the target signal light is lit in red or yellow and the vehicle's speed exceeds a threshold.

11. A vehicle, characterized in that, The vehicle is equipped with a driving assistance device as described in any one of claims 1 to 10.

12. A driving assistance method, characterized in that, The driving assistance method includes: The shooting step involves shooting a picture of the front of the vehicle. The determination step involves identifying the signal lights within the image obtained through the shooting step. The detection step involves detecting, based on the image, the setting height, the lateral distance from the vehicle, and the travel direction distance from the vehicle for the traffic light determined by the determination step. The determination step involves determining, based on the detection result of the detection step, whether the signal light determined by the determination step is an object signal light indicating whether the vehicle can proceed; and The alarm output step involves outputting an alarm to the driver. When multiple traffic lights are identified from the image through the determination step, in the detection step, for each of the multiple traffic lights, the setting height, the lateral distance, and the travel direction distance are detected based on the image. In the judgment step, based on the detection results of the detection step, the traffic light among the multiple traffic lights that has the setting height that meets the predetermined condition and the shortest lateral distance is judged as the first candidate of the target traffic light, and the traffic light that has the setting height that meets the predetermined condition and the shortest travel direction distance is judged as the second candidate of the target traffic light. In the alarm output step, the alarm is output based on the combination of the lighting status of the first candidate traffic light and the lighting status of the second candidate traffic light.

13. A storage medium, wherein, The storage medium stores a program for causing a computer to execute the driving assistance method of claim 12.

Images