Traffic light identification device for vehicle, vehicle, traffic light identification method and program for vehicle

Abstract

To appropriately identify a traffic light for a self-vehicle which the self-vehicle has to obey from among a plurality of traffic lights included in a front camera image.SOLUTION: A traffic light identification device 20 for a self-vehicle performs: identifying a driver gazing position which corresponds to a position gazed by a driver in front of a self-vehicle 1 and appears on a front camera image captured by a front camera 2 based on a positional relation between a driver monitor camera 7 and the front camera 2 and also based on a driver monitor camera image including the face of the driver of the self-vehicle 1 captured by the driver monitor camera 7; detecting traffic lights included in the front camera image; identifying the traffic light for the self-vehicle which the self-vehicle has to obey among a plurality of traffic lights included in the front camera image when the plurality of traffic lights are detected; and identifying the traffic light located at the driver gazing position among the traffic lights included in the front camera image as the traffic light for the self-vehicle when there is no preceding vehicle in front of the self-vehicle, the self-vehicle is entering an intersection, and also a braking operation by the driver is detected.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a traffic light identification device for a vehicle, a vehicle, a traffic light identification method for a vehicle, and a program. [Background technology]

[0002] Patent Document 1 describes a technology that provides driving assistance to a driver of a vehicle entering an intersection. In the technology described in Patent Document 1, when a driver of a vehicle entering an intersection attempts to turn right, for example, at the intersection, a vehicle driving assistance system determines whether the vehicle can pass through the intersection in accordance with a right turn permission signal, or whether the vehicle needs to stop at the intersection in accordance with a stop signal before passing through the intersection in accordance with the right turn permission signal, and notifies the driver of the determination result. However, the technology described in Patent Document 1 does not use a front camera image captured by a front camera mounted on the vehicle to identify the traffic light the vehicle should obey. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-166396 Summary of the Invention [Problem to be solved by the invention]

[0004] The front camera image may include traffic lights other than the one that the vehicle should follow. For example, in order to provide driving assistance based on the front camera image, such as informing the driver that the display signal of the traffic light that the vehicle should follow has changed, a technology is required that can appropriately identify the traffic light that the vehicle should follow from the multiple traffic lights included in the front camera image.

[0005] In view of the above, the present disclosure aims to provide a vehicle traffic light identification device, a vehicle, a vehicle traffic light identification method, and a program that can appropriately identify the vehicle traffic light that the vehicle should follow from multiple traffic lights included in a forward camera image. [Means for solving the problem]

[0006] (1) One aspect of the present disclosure includes a gaze position specifying unit that specifies a driver gaze position, which is a position on a front camera image captured by the front camera corresponding to a position in front of the host vehicle where the driver is gazing, based on a positional relationship between a driver monitor camera and a front camera and a driver monitor camera image including the face of the driver of the host vehicle captured by the driver monitor camera; a traffic light detection unit that detects traffic lights included in the front camera image; and a host vehicle signal detector that is a traffic light that the host vehicle should follow among the plurality of traffic lights when multiple traffic lights included in the front camera image are detected by the traffic light detection unit. a preceding vehicle determination unit that determines whether a preceding vehicle is present in front of the host vehicle; and a braking operation detection unit that detects a braking operation by the driver, wherein the host vehicle traffic light identification unit identifies a traffic light that is present at the driver's gaze position among the plurality of traffic lights included in the forward camera image as the host vehicle traffic light when there is no preceding vehicle present in front of the host vehicle, the host vehicle is entering the intersection, and the braking operation by the driver is detected.

[0007] (2) In the vehicle equipped with the traffic light identification device for the vehicle and the warning device of (1), the warning device may include a signal determination unit that determines whether the display of the traffic light for the vehicle has changed from red to green or green, an accelerator pedal determination unit that determines whether the accelerator pedal has been depressed by the driver before a predetermined time has elapsed since the time the display of the traffic light for the vehicle has changed from red to green or green, and a control unit that executes processing to output a warning if the driver has not depressed the accelerator pedal before the predetermined time has elapsed since the time the display of the traffic light for the vehicle has changed from red to green or green.

[0008] (3) In the subject vehicle equipped with the subject vehicle traffic light identification device and communication device of (1), the communication device transmits data of the forward camera image and information indicating the subject vehicle traffic light contained in the forward camera image identified by the subject vehicle traffic light identification unit to a server device, and the server device performs machine learning of a machine learning model that estimates a traffic light for an autonomous vehicle that is contained in an image captured by a forward camera mounted on the autonomous vehicle and is a traffic light that the autonomous vehicle should follow, and the data of the forward camera image transmitted by the communication device and the information indicating the subject vehicle traffic light contained in the forward camera image identified by the subject vehicle traffic light identification unit may be used as correct answer data in the machine learning of the machine learning model.

[0009] (4) One aspect of the present disclosure is a method for detecting a traffic light for a host vehicle, the method comprising: a gaze position identification step in which a traffic light identification device for a host vehicle identifies a driver's gaze position, which is a position on a front camera image captured by the front camera, corresponding to a position in front of the host vehicle where the driver is gazing, based on a positional relationship between a driver monitor camera and a front camera and a driver monitor camera image including the face of the driver of the host vehicle captured by the driver monitor camera; a traffic light detection step in which the traffic light identification device for the host vehicle detects a traffic light included in the front camera image; and when a plurality of traffic lights included in the front camera image are detected in the traffic light detection step, the traffic light identification device for the host vehicle identifies a traffic light for the host vehicle that is a traffic light that the host vehicle should follow from among the plurality of traffic lights. a preceding vehicle determination step in which the traffic light identification device for the host vehicle determines whether the host vehicle is entering an intersection; a preceding vehicle determination step in which the traffic light identification device for the host vehicle determines whether a preceding vehicle is present in front of the host vehicle; and a braking operation detection step in which the traffic light identification device for the host vehicle detects a braking operation by the driver, wherein in the traffic light identification step for the host vehicle, when there is no preceding vehicle in front of the host vehicle, the host vehicle is entering the intersection, and the braking operation by the driver is detected, a traffic light that is present at the driver's gaze position among the multiple traffic lights included in the forward camera image is identified as the traffic light for the host vehicle.

[0010] (5) One aspect of the present disclosure is a method for detecting a driver's gaze position, which is a position on a front camera image captured by the front camera that corresponds to a position in front of the host vehicle where the driver is gazing, based on a positional relationship between a driver monitor camera and a front camera and a driver monitor camera image including the face of the driver of the host vehicle captured by the driver monitor camera; a traffic light detection step for detecting a traffic light included in the front camera image; and, when a plurality of traffic lights included in the front camera image are detected in the traffic light detection step, a traffic light for the host vehicle that is a traffic light that the host vehicle should follow from among the plurality of traffic lights. a preceding vehicle determination step for determining whether a preceding vehicle is present in front of the host vehicle; and a braking operation detection step for detecting a braking operation by the driver, wherein in the traffic light identification step for the host vehicle, when there is no preceding vehicle in front of the host vehicle, the host vehicle is entering the intersection, and the braking operation by the driver is detected, the traffic light that is present at the driver's gaze position among the multiple traffic lights included in the forward camera image is identified as the traffic light for the host vehicle. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to appropriately identify the traffic light for the host vehicle that the host vehicle should follow from among multiple traffic lights included in a forward camera image. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing an example of a schematic configuration of a host vehicle 1 to which a host vehicle traffic light identification device 20 according to a first embodiment is applied. [Figure 2] 2 is a diagram showing an example of a specific configuration of a traffic light identification device 20 for a vehicle shown in FIG. 1. FIG. [Figure 3] 2 is a diagram showing an example of a specific configuration of the alarm device 30 shown in FIG. 1. FIG. [Figure 4]FIG. 4 is a diagram showing an example of a front camera image G2 including a plurality of traffic lights T01 to T04 captured by the front camera 2. As shown in FIG. [Figure 5] 10 is a diagram showing an example of a driver's gaze position G2P on a front camera image G2 identified by a gaze position identifying unit 232. FIG. [Figure 6] 10 is a diagram showing another example of the driver's gaze position G2P on the front camera image G2 identified by the gaze position identifying unit 232. FIG. [Figure 7] 4 is a flowchart illustrating an example of processing executed by a processor 23 of the traffic light identification device 20 for the vehicle of the first embodiment. [Figure 8] 4 is a flowchart illustrating an example of processing executed by a processor 33 of the alarm device 30 shown in FIG. [Figure 9] 10 is a diagram showing an example of a data collection system SM including a host vehicle 1 to which a traffic light identification device 20 for a host vehicle according to a second embodiment is applied, and a server device SV. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, with reference to the drawings, embodiments of a traffic light identification device for a vehicle, a vehicle, a traffic light identification method for a vehicle, and a program according to the present disclosure will be described.

[0014] First Embodiment Fig. 1 is a diagram showing an example of a schematic configuration of a host vehicle 1 to which a host vehicle traffic light identification device 20 according to a first embodiment is applied. Fig. 2 is a diagram showing an example of a specific configuration of the host vehicle traffic light identification device 20 shown in Fig. 1. Fig. 3 is a diagram showing an example of a specific configuration of the warning device 30 shown in Fig. 1. In the example shown in FIGS. 1 to 3 , the host vehicle 1 is equipped with a front camera 2, a radar 3, a LiDAR (Light Detection And Ranging) 4, a driving assistance ECU (Electronic Control Unit) 12, a host vehicle traffic light identification device 20, and an alarm device 30. The front camera 2 captures images of surrounding vehicles (e.g., a preceding vehicle) and the road environment (e.g., traffic lights, road structure, rules, etc.) around the host vehicle 1, generates image data of the surrounding vehicles and the surrounding road environment, and transmits the image data to the driving assistance ECU 12, the host vehicle traffic light identification device 20, and the alarm device 30. The radar 3 is, for example, a millimeter-wave radar, and detects the relative positions and relative speeds of the surrounding vehicles and surrounding road structure with respect to the host vehicle 1, and transmits the detection results to the driving assistance ECU 12 and the alarm device 30. The LiDAR 4 detects the relative positions and relative speeds of the surrounding vehicles and surrounding road structure with respect to the host vehicle 1, and transmits the detection results to the driving assistance ECU 12.

[0015] The host vehicle 1 also includes a GPS (Global Positioning System) unit 5 and a map information unit 6. The GPS unit 5 acquires location information indicating the current location of the host vehicle 1 based on a GPS signal, and transmits the location information of the host vehicle 1 to the driving assistance ECU 12. The map information unit 6 is formed in a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) mounted on the host vehicle 1. The map information stored in the map information unit 6 includes various information such as road structure (road location, road shape, lane structure, etc.), rules, etc. Furthermore, the host vehicle 1 is equipped with a driver monitor camera 7 and an HMI (Human Machine Interface) 8. The driver monitor camera 7 captures a driver monitor camera image including the face of the driver of the host vehicle 1. The driver monitor camera 7 is disposed above the steering column (not shown) of the host vehicle 1 so as to be able to capture an image of the face and part of the upper body of the driver of the host vehicle 1. In other examples, the driver monitor camera 7 may be disposed in the center cluster of the host vehicle 1, or the driver monitor camera 7 may be disposed in the rearview mirror, meter panel, meter hood, etc. of the host vehicle 1. In these examples, the driver monitor camera 7 can also capture an image of the face and part of the upper body of the driver of the host vehicle 1.

[0016] 1 to 3, the HMI 8 is an interface for inputting and outputting information between the driver and the driving assistance ECU 12, warning device 30, etc. The HMI 8 is an information providing device for providing various information to the driver, specifically, it is equipped with a display for displaying characters, images, etc., a speaker for outputting sound, etc. The HMI 8 also has operation buttons, a touch panel, etc. for receiving input operations from the driver. The host vehicle 1 is also equipped with a brake switch 9 and an accelerator pedal sensor 10. The brake switch 9 detects whether or not the driver of the host vehicle 1 is operating a brake pedal (not shown), and transmits the detection result to the host vehicle traffic light identification device 20. The accelerator pedal sensor 10 is also referred to as an accelerator operation amount sensor or accelerator position sensor, for example, and detects the amount of operation of the accelerator pedal (not shown) by the driver of the host vehicle 1, and transmits the detection result to the warning device 30. The forward camera 2, radar 3, LiDAR 4, GPS unit 5, map information unit 6, driver monitor camera 7, HMI 8, brake switch 9, accelerator pedal sensor 10, driving assistance ECU 12, vehicle traffic light identification device 20, and warning device 30 are connected via an in-vehicle network 13.

[0017] The host vehicle 1 also includes a steering actuator 14, a braking actuator 15, and a drive actuator 16. The steering actuator 14 includes, for example, a power steering system, a steer-by-wire steering system, a rear-wheel steering system, etc. The braking actuator 15 has a function of decelerating the host vehicle 1. The braking actuator 15 includes, for example, a hydraulic brake, a regenerative brake, etc. The drive actuator 16 has a function of accelerating the host vehicle 1. The drive actuator 16 includes, for example, an engine, an EV (electric vehicle) system, a hybrid system, a fuel cell system, etc. The driving assistance ECU 12 transmits a control signal to the steering actuator 14 for activating the steering actuator 14, transmits a control signal to the braking actuator 15 for activating the braking actuator 15, and transmits a control signal to the driving actuator 16 for activating the driving actuator 16. The traffic light identification device 20 for the host vehicle identifies a traffic light TX for the host vehicle 1 that is a traffic light that the host vehicle 1 should follow from among a plurality of traffic lights included in the front camera image G2 captured by the front camera 2.

[0018] FIG. 4 is a diagram showing an example of a front camera image G2 including a plurality of traffic lights T01 to T04 captured by the front camera 2. As shown in FIG. In the example shown in Fig. 4, four traffic lights T01 to T04 are included in the front camera image G2. Traffic lights T01 and T02 are displaying a red light, and traffic lights T03 and T04 are displaying a green light (or a green light). Of the four traffic lights T01 to T04, traffic lights T01 and T02 are host vehicle traffic lights TX that the host vehicle 1 should follow, and traffic lights T03 and T04 are traffic lights that the host vehicle 1 should not follow. In other words, the situation shown in Fig. 4 is one in which the host vehicle 1 needs to stop in front of the stop line SL (lower side of Fig. 4) in accordance with the host vehicle traffic light TX (traffic light T01 or traffic light T02) that is displaying a red light. If the vehicle traffic light identification device 20 identifies either traffic light T03 or traffic light T04 as the vehicle traffic light TX without identifying either traffic light T01 or traffic light T02, there is a risk that driving assistance will not be provided appropriately to the driver of the vehicle 1. In consideration of this point, the traffic light identification device 20 for the vehicle of the first embodiment has the measures described below implemented so that the traffic light identification device 20 for the vehicle can appropriately identify the traffic light TX for the vehicle (traffic light T01 or traffic light T02) that the vehicle 1 should follow from the multiple traffic lights T01 to T04 included in the forward camera image G2.

[0019] In the example shown in Figures 1 to 3, the host vehicle traffic light identification device 20 is configured by a microcomputer including a communication interface (I / F) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the host vehicle traffic light identification device 20 to the in-vehicle network 13. The memory 22 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores programs and various data used in the processing executed by the processor 23. The processor 23 has functions such as identifying the host vehicle traffic light TX included in the forward camera image G2. 1 to 3, the host vehicle traffic light identification device 20 includes one processor 23, but in other examples, the host vehicle traffic light identification device 20 may include multiple processors. Also, in the example shown in Figures 1 to 3, the host vehicle traffic light identification device 20 is configured with one ECU (e.g., the host vehicle traffic light identification ECU), but in other examples, the host vehicle traffic light identification device 20 may be configured with multiple ECUs (e.g., the host vehicle traffic light identification ECU, a driving assistance ECU, an alarm ECU, etc.).

[0020] In the example shown in Figures 1 to 3, the processor 23 has a function as an acquisition unit 231, a function as a gaze position identification unit 232, a function as a traffic light detection unit 233, a function as a traffic light identification unit for the vehicle itself 234, a function as an intersection entry determination unit 235, a function as a preceding vehicle determination unit 236, and a function as a braking operation detection unit 237. The acquisition unit 231 acquires data of a front camera image G2 including a plurality of traffic lights captured by the front camera 2 from the front camera 2. The acquisition unit 231 also acquires data of a driver monitor camera image including the face of the driver of the vehicle 1 captured by the driver monitor camera 7 from the driver monitor camera 7. Furthermore, the acquisition unit 231 acquires information indicating the positional relationship between the front camera 2 and the driver monitor camera 7 from the memory 22, for example. The gaze position identifying unit 232 identifies a driver gaze position G2P, which is a position on the front camera image G2 captured by the front camera 2, corresponding to the position in front of the host vehicle 1 that the driver of the host vehicle 1 is gazing at, based on the positional relationship between the driver monitor camera 7 and the front camera 2 and the driver monitor camera image including the face of the driver of the host vehicle 1 captured by the driver monitor camera 7. The gaze position identifying unit 232 identifies the driver gaze position G2P on the front camera image G2 by using, for example, the technology described in paragraphs 0018 and 0019 of Japanese Patent No. 6459856.

[0021] FIG. 5 is a diagram showing an example of the driver's gaze position G2P on the front camera image G2 identified by the gaze position identifying unit 232. As shown in FIG. In the example shown in FIG. 5, five traffic lights T05 to T09 are included in the front camera image G2. Traffic lights T05, T06, and T07 are displaying red lights, and traffic lights T08 and T09 are displaying green lights (or green lights). Of the five traffic lights T05 to T09, traffic lights T05 and T06 are the traffic lights that the host vehicle 1 should obey, and the situation shown in FIG. 5 is one in which the host vehicle 1 needs to stop before the stop line SL (lower side of FIG. 5) in accordance with traffic light T05 or traffic light T06 that is displaying red lights. In other words, traffic lights T07, T08, and T09 are traffic lights installed at an intersection different from the intersection where traffic lights T05 and T06 are installed (i.e., the intersection that the host vehicle 1 is about to pass through), and are traffic lights that the host vehicle 1 should not obey. 5, the driver of the host vehicle 1 is gazing at the red light displayed by traffic light T05 and is trying to stop the host vehicle 1 in front of the stop line SL in accordance with the red light. Therefore, the gaze position identification unit 232 identifies the driver's gaze position G2P on the front camera image G2 captured by the front camera 2, which corresponds to the position in front of the host vehicle 1 that the driver of the host vehicle 1 is gazing at (more specifically, the position of traffic light T05), based on the positional relationship between the driver monitor camera 7 and the front camera 2 and the driver monitor camera image including the face of the driver of the host vehicle 1 captured by the driver monitor camera 7.

[0022] 1 to 3, the traffic light detection unit 233 detects traffic lights included in the front camera image G2 captured by the front camera 2. The traffic light detection unit 233 detects traffic lights included in the front camera image G2 by using, for example, the technology described in paragraph 0022 of Japanese Patent No. 7073880. The host vehicle traffic light identification unit 234 identifies a host vehicle traffic light TX that is a traffic light that the host vehicle 1 should follow from among the multiple traffic lights T01 to T04 (see FIG. 4) included in the forward camera image G2 when the traffic light detection unit 233 detects them. In more detail, the host vehicle traffic light identification unit 234 identifies the host vehicle traffic light TX by using a method that will be described later.

[0023] The intersection entry determination unit 235 determines whether or not the vehicle 1 is entering an intersection. In detail, the intersection entry determination unit 235 determines whether or not the vehicle 1 is entering an intersection based on the position information of the vehicle 1 acquired by the GPS unit 5 and the map information held by the map information unit 6. In another example, the intersection entry determination unit 235 may determine whether the vehicle 1 is entering an intersection based on the forward camera image G2 captured by the forward camera 2, for example, by using the technology described in paragraphs 0068 to 0070 of Japanese Patent No. 7095638.

[0024] 1 to 3, the preceding vehicle determination unit 236 determines whether or not there is a preceding vehicle ahead of the host vehicle 1 (more specifically, a preceding vehicle located between the host vehicle 1 and the intersection and traveling in the same lane as the host vehicle 1 is traveling in). The preceding vehicle determination unit 236 determines whether or not there is a preceding vehicle ahead of the host vehicle 1 based on the front camera image G2 captured by the front camera 2 and the detection result of the radar 3, similar to the technology described in paragraph 0043 of Japanese Patent No. 7176467, for example. In another example, the preceding vehicle determination unit 236 may determine whether or not a preceding vehicle is present ahead of the host vehicle 1 based solely on the detection result of the radar 3.

[0025] 1 to 3, the braking operation detection unit 237 detects a braking operation by the driver of the host vehicle 1. In detail, the braking operation detection unit 237 detects a braking operation by the driver of the host vehicle 1 based on an output signal of the brake switch 9 (i.e., a signal indicating whether or not the driver of the host vehicle 1 has operated the brake pedal). In another example, the braking operation detection unit 237 may detect a braking operation by the driver of the vehicle 1 based on, for example, the detection result of a master cylinder pressure sensor (not shown).

[0026] 1 to 3, when the preceding vehicle determination unit 236 determines that there is no preceding vehicle ahead of the host vehicle 1, when the intersection entry determination unit 235 determines that the host vehicle 1 is entering an intersection, and when the braking operation detection unit 237 detects a braking operation by the driver of the host vehicle 1, the host vehicle traffic light identification unit 234 identifies, for example, traffic light T05 that is located at the driver's gaze position G2P among the multiple traffic lights T05 to T09 included in the forward camera image G2 shown in Fig. 5 as the host vehicle traffic light TX. This is because the driver of the host vehicle 1 starts a braking operation after visually recognizing the red signal of the host vehicle traffic light TX, and stops the host vehicle 1 on the near side of the stop line SL.

[0027] FIG. 6 is a diagram showing another example of the driver's gaze position G2P on the front camera image G2 identified by the gaze position identifying unit 232. In FIG. In the example shown in FIG. 6, three traffic lights T10 to T12 are included in the front camera image G2. Traffic light T10 is displaying a red light, and traffic lights T11 and T12 are displaying a green light (or a green light). Of the three traffic lights T10 to T12, traffic light T10 is the traffic light that the host vehicle 1 should obey, and the situation shown in FIG. 6 is a situation in which the host vehicle 1 needs to stop before the stop line SL (lower side of FIG. 6) in accordance with traffic light T10, which is displaying a red light. In other words, traffic lights T11 and T12 are traffic lights installed at an intersection different from the intersection where traffic light T10 is installed (i.e., the intersection that the host vehicle 1 is about to pass through), and are traffic lights that the host vehicle 1 should not obey. 6, the driver of the host vehicle 1 is gazing at a red light displayed by traffic light T10 and is trying to stop the host vehicle 1 in front of the stop line SL in accordance with the red light. Therefore, the gaze position identification unit 232 identifies the driver's gaze position G2P on the front camera image G2 captured by the front camera 2, which corresponds to the position in front of the host vehicle 1 that the driver of the host vehicle 1 is gazing at (more specifically, the position of traffic light T10), based on the positional relationship between the driver monitor camera 7 and the front camera 2 and the driver monitor camera image including the face of the driver of the host vehicle 1 captured by the driver monitor camera 7.

[0028] In the example shown in Fig. 6, the intersection entry determination unit 235 determines that the host vehicle 1 is entering an intersection where traffic light T10 is installed. Furthermore, the preceding vehicle determination unit 236 determines that there is no preceding vehicle ahead of the host vehicle 1 (more specifically, a preceding vehicle located between the host vehicle 1 and the intersection where traffic light T10 is installed and traveling in the same lane as the host vehicle 1). Furthermore, the braking operation detection unit 237 detects a braking operation by the driver of the host vehicle 1. As a result, the host vehicle traffic light identification unit 234 identifies the traffic light T10 located at the driver's gaze position G2P from among the multiple traffic lights T10 to T12 included in the forward camera image G2 shown in Fig. 6 as the host vehicle traffic light TX.

[0029] In the example shown in FIGS. 1 to 3, the warning device 30 is configured by a microcomputer including a communication interface 31, a memory 32, and a processor 33. The communication interface 31, the memory 32, and the processor 33 are connected via a signal line 34. The communication interface 31 has an interface circuit for connecting the warning device 30 to the in-vehicle network 13. The memory 32 stores programs and various data used in the processing executed by the processor 33. The processor 23 has a function of notifying the driver of the vehicle 1 that the display of the host vehicle's traffic light TX included in the front camera image G2 has changed from a red light to a green light (or a green light), etc. The processor 33 has a function as a signal determination unit 331 , a function as an accelerator pedal determination unit 332 , and a function as a control unit 333 .

[0030] The signal determination unit 331 determines whether the display of the host vehicle's traffic light TX included in the front camera image G2 has changed from a red light to a green light (or a green light). The signal determination unit 331 determines whether the display of the host vehicle's traffic light TX included in the front camera image G2 has changed from a red light to a green light (or a green light) by using, for example, the technology described in the specification of U.S. Patent Application Publication No. 2013 / 0253754 or the technology described in paragraph 0057 of Japanese Patent No. 7180565. The accelerator pedal determination unit 332 determines whether or not the driver of the host vehicle 1 has depressed the accelerator pedal before a predetermined time has elapsed since the display of the host vehicle's traffic light TX included in the forward camera image G2 changed from a red light to a green light (or a green light). More specifically, the accelerator pedal determination unit 332 determines whether or not the driver of the host vehicle 1 has depressed the accelerator pedal based on the detection result of the accelerator pedal sensor 10. The control unit 333 executes a process to output an alarm to the HMI 8 to prompt the driver of the vehicle 1 to start the vehicle 1 if the driver of the vehicle 1 does not press the accelerator pedal before a predetermined time has elapsed from the time when the display of the vehicle's traffic light TX included in the forward camera image G2 changes from red to green (or green).

[0031] FIG. 7 is a flowchart illustrating an example of processing executed by the processor 23 of the traffic light identification device 20 for the vehicle according to the first embodiment. In the example shown in FIG. 7, the acquisition unit 231 acquires data of the front camera image G2, data of the driver monitor camera image, and information indicating the positional relationship between the front camera 2 and the driver monitor camera 7 in step S10. In step S11, the gaze position identification unit 232 identifies the driver gaze position G2P, which is the position on the forward camera image G2 captured by the forward camera 2 that corresponds to the position in front of the vehicle 1 that the driver of the vehicle 1 is gazing at, based on the positional relationship between the forward camera 2 and the driver monitor camera 7 and the driver monitor camera image. In step S12, the traffic light detection unit 233 detects a traffic light included in the front camera image G2.

[0032] In step S13, the intersection entry determination unit 235 determines whether or not the vehicle 1 is entering an intersection. If YES, the process proceeds to step S14, and if NO, the routine shown in FIG. In step S14, the preceding vehicle determination unit 236 determines whether or not a preceding vehicle is present ahead of the host vehicle 1. If NO, the process proceeds to step S15, and if YES, the routine shown in FIG. 7 is terminated. In step S15, for example, the host vehicle traffic light identification unit 234 determines whether or not the braking operation detection unit 237 has detected a braking operation by the driver of the host vehicle 1. If YES, the process proceeds to step S16, and if NO, the routine shown in FIG. In step S16, the host vehicle traffic light identification unit 234 identifies a host vehicle traffic light TX that should be followed by the host vehicle 1 from among the traffic lights included in the front camera image G2 detected in step S12. In detail, the host vehicle traffic light identification unit 234 identifies, from among the traffic lights included in the front camera image G2 detected in step S12, the traffic light that is located at the driver's gaze position G2P on the front camera image G2 identified in step S11 as the host vehicle traffic light TX.

[0033] FIG. 8 is a flowchart illustrating an example of processing executed by the processor 33 of the alarm device 30 shown in FIG. 8, in step S20, the signal determination unit 331 determines whether the display of the host vehicle traffic light TX included in the front camera image G2 has changed from a red light to a green light (or a green light). If the result is YES, the process proceeds to step S21, and if the result is NO, the routine shown in FIG. 8 is terminated. In step S21, the accelerator pedal determination unit 332 determines whether the accelerator pedal is depressed by the driver of the vehicle 1 before a predetermined time has elapsed since the display of the vehicle's traffic light TX included in the forward camera image G2 changed from red to green (or green). If the result is NO, the process proceeds to step S22, and if the result is YES, the routine shown in FIG. 8 is terminated. In step S22, the control unit 333 executes a process of causing the HMI 8 to output an alarm that prompts the driver of the host vehicle 1 to start the host vehicle 1.

[0034] Second Embodiment The host vehicle 1 to which the host vehicle traffic light identification device 20 of the second embodiment is applied is configured similarly to the host vehicle 1 to which the host vehicle traffic light identification device 20 of the first embodiment shown in Fig. 1 is applied, except for the points described below. Moreover, the host vehicle traffic light identification device 20 of the second embodiment is configured similarly to the host vehicle traffic light identification device 20 of the first embodiment shown in Fig. 2.

[0035] FIG. 9 is a diagram showing an example of a data collection system SM including a host vehicle 1 to which a traffic light identification device 20 for a host vehicle according to the second embodiment is applied, and a server device SV. As described above, in the example shown in FIG. 9, the host vehicle 1 does not include a warning device 30 but includes a communication device 40. The communication device 40 is configured, for example, by a DCM (Data Communication Module). The communication device 40 has a function of transmitting data of a front camera image G2 including a plurality of traffic lights captured by the front camera 2 to the server device SV. The communication device 40 also has a function of transmitting information indicating the host vehicle's traffic light TX included in the front camera image G2 identified by the host vehicle's traffic light identification unit 234 to the server device SV.

[0036] 9, the server device SV is configured by a computer having, for example, a CPU (Central Processing Unit), memory, storage, a NIC (Network Interface Card), etc. The server device SV includes a communication unit (not shown), a storage unit (not shown), and a control unit (not shown). For example, the NIC functions as a communication unit of the server device SV. The communication unit of the server device SV receives data of the front camera image G2 transmitted by the communication device 40 of the host vehicle 1 and information indicating the host vehicle's traffic light TX included in the front camera image G2. For example, memories such as RAM (Random Access Memory) and ROM (Read Only Memory) and storages such as HDD and SSD function as storage units of the server device SV. The storage unit of the server device SV stores data of the front camera image G2 received by the communication unit of the server device SV and information indicating the traffic light TX for the host vehicle included in the front camera image G2. For example, the CPU functions as a control unit of the server device SV. The control unit of the server device SV executes, for example, a process of causing the communication unit of the server device SV to receive data of the front camera image G2 transmitted by the communication device 40 of the host vehicle 1 and information indicating the host vehicle's traffic light TX included in the front camera image G2, and a process of storing the data of the front camera image G2 received by the communication unit of the server device SV and information indicating the host vehicle's traffic light TX included in the front camera image G2 in a storage unit of the server device SV. Furthermore, the control unit of the server device SV executes machine learning of a machine learning model that estimates a traffic light for an autonomous vehicle, which is a traffic light included in an image captured by a front camera (not shown) mounted on the autonomous vehicle (not shown) and which the autonomous vehicle should follow. The data of the front camera image G2 transmitted by the communication device 40 of the host vehicle 1 and received by the communication unit of the server device SV and the information indicating the host vehicle's traffic light TX included in the front camera image G2 are used as correct answer data for the machine learning of the machine learning model.

[0037] As described above, in the example shown in Figures 1 to 3, when the driver of vehicle 1 gazes at the red light displayed by a traffic light and attempts to stop vehicle 1 in front of the stop line in accordance with the red light, the gaze position identification unit 232 identifies the driver's gaze position G2P on the forward camera image G2 captured by the forward camera 2, which corresponds to the position in front of vehicle 1 that the driver of vehicle 1 is gazing at (more specifically, the position of the traffic light displaying the red light), based on the positional relationship between the driver monitor camera 7 and the forward camera 2 and the driver monitor camera image including the face of the driver of vehicle 1 captured by the driver monitor camera 7. 9, similar to the examples shown in FIGS. 1 to 3, when the driver of the host vehicle 1 gazes at a red light displayed by a traffic light and attempts to stop the host vehicle 1 before the stop line, the gaze position identifying unit 232 identifies the driver's gaze position G2P on the front camera image G2 captured by the front camera 2. Furthermore, the communication device 40 of the host vehicle 1 transmits data on the front camera image G2 captured by the front camera 2 and information indicating the host vehicle's traffic light TX (the traffic light displaying a red light that is located at the driver's gaze position G2P on the front camera image G2) included in the front camera image G2 to the server device SV as supervised data to be used in the machine learning of the machine learning model.

[0038] 9, after the traffic light changes from red to green, when the driver of the host vehicle 1 gazes at the green light displayed by the traffic light and restarts the host vehicle 1 in accordance with the green light, the gaze position identification unit 232 identifies the driver's gaze position G2P on the front camera image G2 captured by the front camera 2. Furthermore, the communication device 40 of the host vehicle 1 transmits data on the front camera image G2 captured by the front camera 2 and information indicating the host vehicle's traffic light TX (the traffic light that is displaying a green light and is located at the driver's gaze position G2P on the front camera image G2) included in the front camera image G2 to the server device SV as supervised data to be used in the machine learning of the machine learning model.

[0039] As a background to the traffic light identification device 20 for a vehicle according to the first and second embodiments, there is a demand for the application of autonomous driving (driving assistance) technology to public roads where intersections including traffic lights exist. A vehicle driver stops the vehicle at a red light at an intersection and restarts the vehicle when the light changes from red to green. To realize the application of autonomous driving technology to public roads, a technology is needed that identifies a traffic light that the vehicle should follow from multiple traffic lights included in a front camera image captured by a front camera mounted on the vehicle. In particular, to realize the application of autonomous driving technology to public roads, a technology is needed that identifies a traffic light that the vehicle should follow using only a front camera image captured by a front camera mounted on the vehicle (i.e., without the need to use position information of the vehicle, etc.).

[0040] In order to perform machine learning on a machine learning model that estimates which traffic lights contained in images captured by a forward-facing camera (not shown) mounted on an autonomous vehicle (not shown) should be obeyed by the autonomous vehicle, a huge amount of correct answer data is required. Furthermore, if the traffic light that the vehicle must obey is a special traffic light, such as a roadside traffic light typical of Europe, or a local traffic light specific to a particular area set by a local government, it will be difficult for the annotator to identify the traffic light that the vehicle must obey and create correct answer data using only the forward camera image captured by the forward camera mounted on the vehicle, and an increase in the annotator's work costs is expected.

[0041] As described above, in the host vehicle 1 equipped with the host vehicle traffic light identification device 20 of the second embodiment, when a front camera image G2 is captured by the front camera 2 and multiple traffic lights are included in the front camera image G2, the host vehicle traffic light TX that the host vehicle 1 should follow from the multiple traffic lights included in the front camera image G2 is identified by the host vehicle traffic light identification unit 234. In other words, the host vehicle traffic light identification unit 234 performs a process of identifying the host vehicle traffic light TX, which is difficult for an annotator to identify. Therefore, by using the host vehicle traffic light identification device 20 of the second embodiment, it is possible to improve the applicability of autonomous driving technology to general roads where intersections including traffic lights exist.

[0042] Furthermore, as described above, in the host vehicle 1 equipped with the host vehicle traffic light identification device 20 of the first embodiment, when a front camera image G2 is captured by the front camera 2 and the front camera image G2 includes multiple traffic lights, the host vehicle traffic light TX that the host vehicle 1 should follow from the multiple traffic lights included in the front camera image G2 is identified by the host vehicle traffic light identification unit 234. Therefore, in the host vehicle 1 equipped with the host vehicle traffic light identification device 20 of the first embodiment, when the display of the host vehicle traffic light TX changes from a red light to a green light (or a green light), the driver of the host vehicle 1 can be prompted to start the host vehicle 1 based on the front camera image G2 that includes the host vehicle traffic light TX.

[0043] As described above, the embodiments of the traffic light identification device for a vehicle, the vehicle, the traffic light identification method for a vehicle, and the program of the present disclosure have been described with reference to the drawings, but the traffic light identification device for a vehicle, the vehicle, the traffic light identification method, and the program of the present disclosure are not limited to the above-described embodiments, and appropriate modifications can be made without departing from the spirit of the present disclosure. The configurations of the examples of the above-described embodiments can also be combined as appropriate. In each of the above-described embodiments, the processing performed in the vehicle-use traffic light identification device 20 has been described as software processing performed by executing a program, but the processing performed in the vehicle-use traffic light identification device 20 may be processing performed by hardware. Alternatively, the processing performed in the vehicle-use traffic light identification device 20 may be processing that combines both software and hardware. Furthermore, the program stored in the memory 22 of the vehicle-use traffic light identification device 20 (a program that realizes the functions of the processor 23 of the vehicle-use traffic light identification device 20) may be provided, distributed, or the like, recorded on a computer-readable storage medium such as a semiconductor memory, a magnetic recording medium, an optical recording medium, or the like. [Explanation of symbols]

[0044] 1. Your vehicle 2. Front camera 3. Radar 4. LiDAR 5 GPS units 6 Map Information Unit 7 Driver monitor camera 8 HMI 9. Brake switch 10 Accelerator pedal sensor 12 Driver assistance ECU 14 Steering actuator 15 Braking Actuator 16 Drive Actuator 20. Traffic light identification device for vehicle 21 Communication Interface 22 Memory 23 processors 231 Acquisition Department 232 Gaze position identification unit 233 Traffic light detector 234 Vehicle signal identification unit 235 Intersection approach determination section 236 Leading vehicle detection unit 237 Braking operation detection unit 30 Alarm device 31 Communication Interface 32 memory 33 processors 331 Signal Judgment Unit 332 Accelerator pedal determination unit 333 Control Unit 40 Communication equipment SV server device

Claims

1. a gaze position specifying unit that specifies a driver gaze position, which is a position on the front camera image captured by the front camera, corresponding to a position in front of the vehicle where the driver is gazing, based on a positional relationship between the driver monitor camera and the front camera and a driver monitor camera image including the face of the driver of the vehicle captured by the driver monitor camera; a traffic light detection unit that detects a traffic light included in the front camera image; a subject vehicle traffic light identification unit that identifies a subject vehicle traffic light that is a traffic light that the subject vehicle should follow among a plurality of traffic lights included in the front camera image when the traffic light detection unit detects the plurality of traffic lights; an intersection entry determination unit that determines whether the host vehicle is entering an intersection; a preceding vehicle determination unit that determines whether or not a preceding vehicle is present ahead of the host vehicle; a braking operation detection unit that detects a braking operation by the driver, The traffic light identification unit for the vehicle itself identifies the traffic light that is located at the driver's gaze position among the multiple traffic lights included in the forward camera image as the traffic light for the vehicle itself when there is no preceding vehicle in front of the vehicle, the vehicle is entering the intersection, and the braking operation by the driver is detected.

2. A vehicle equipped with the traffic light identification device for vehicle according to claim 1 and an alarm device, The alarm device a signal determination unit that determines whether the display of the traffic light for the vehicle has changed from a red signal to a green signal or a green signal; an accelerator pedal determination unit that determines whether an accelerator pedal is depressed by the driver before a predetermined time has elapsed since the display of the traffic light for the subject vehicle changed from a red signal to a green signal or a green signal; The vehicle is equipped with a control unit that executes processing to output an alarm if the driver does not press the accelerator pedal before the predetermined time has elapsed from the time the display of the traffic light for the vehicle changes from red to green or green.

3. A vehicle equipped with the traffic light identification device for vehicle according to claim 1 and a communication device, the communication device transmits data of the front camera image and information indicating the traffic light for the host vehicle included in the front camera image identified by the traffic light identification unit for the host vehicle to a server device; the server device executes machine learning of a machine learning model that estimates a traffic light for an autonomous vehicle that is included in an image captured by a forward camera mounted on the autonomous vehicle and that is a traffic light that the autonomous vehicle should follow; The data of the front camera image transmitted by the communication device and the information indicating the traffic light for the vehicle contained in the front camera image identified by the traffic light identification unit for the vehicle are used as correct answer data in the server device for machine learning of the machine learning model for the vehicle.

4. a gaze position specifying step in which the traffic light specifying device for the subject vehicle specifies a driver's gaze position, which is a position on the front camera image captured by the front camera, corresponding to a position in front of the subject vehicle at which the driver is gazing, based on a positional relationship between the driver monitor camera and the front camera and a driver monitor camera image including the face of the driver of the subject vehicle captured by the driver monitor camera; a traffic light detection step in which the traffic light identification device for the vehicle detects a traffic light included in the front camera image; a subject vehicle traffic light identification step in which, when a plurality of traffic lights included in the front camera image are detected in the traffic light detection step, the subject vehicle traffic light identification device identifies a subject vehicle traffic light that is a traffic light that the subject vehicle should follow from among the plurality of traffic lights; an intersection entry determination step in which the traffic light identification device for the host vehicle determines whether the host vehicle is entering an intersection; a preceding vehicle determination step in which the traffic light identification device for the host vehicle determines whether or not a preceding vehicle is present ahead of the host vehicle; a braking operation detection step in which the traffic light identification device for the vehicle detects a braking operation by the driver, In the traffic light identification step for the vehicle, when there is no preceding vehicle in front of the vehicle, the vehicle is entering the intersection, and the braking operation by the driver is detected, the traffic light that is located at the driver's gaze position among the multiple traffic lights included in the forward camera image is identified as the traffic light for the vehicle.

5. The processor a gaze position specifying step of specifying a driver gaze position, which is a position on the front camera image captured by the front camera corresponding to a position in front of the vehicle where the driver is gazing, based on a positional relationship between the driver monitor camera and the front camera and a driver monitor camera image including the face of the driver of the vehicle captured by the driver monitor camera; a traffic light detection step of detecting a traffic light included in the front camera image; a subject vehicle traffic light identification step of identifying a subject vehicle traffic light that is a traffic light that the subject vehicle should follow among the plurality of traffic lights included in the front camera image when the plurality of traffic lights are detected in the traffic light detection step; an intersection entry determination step of determining whether the host vehicle is entering an intersection; a preceding vehicle determination step of determining whether or not a preceding vehicle is present ahead of the host vehicle; a braking operation detection step of detecting a braking operation by the driver, In the traffic light identification step for the vehicle, when there is no preceding vehicle in front of the vehicle, the vehicle is entering the intersection, and the braking operation by the driver is detected, a traffic light that is located at the driver's gaze position among the multiple traffic lights included in the forward camera image is identified as the traffic light for the vehicle.

Images