In-vehicle devices and control programs

Abstract

Stop unnecessary notifications prompting you to start driving. [Solution] The in-vehicle device according to the embodiment has a controller that notifies the driver of the vehicle when the vehicle remains stationary after the traffic light changes from red to green and the vehicle does not start moving. The controller detects the presence or absence of a driver in the driver's seat from the interior image captured by the in-vehicle camera that takes pictures of the inside of the vehicle, and stops outputting the notification when no driver is detected in the driver's seat.

Description

Technical Field

[0001] The present invention relates to an in-vehicle device and a control program.

Background Art

[0002] Conventionally, there is a system that prompts a driver to start when the host vehicle stops at a red traffic light and changes from the red light to a green light. As a technology related to such a system, for example, in a situation where the vehicle hinders running when starting, such as when the host vehicle is a semi-door, a technology for stopping a notification for prompting a start is disclosed (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the prior art does not consider stopping the notification in a situation where the driver is not sitting in the driver's seat, such as when the driver is unloading, and there is room for improvement in stopping unnecessary notifications for prompting a start.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an in-vehicle device and a control program capable of stopping unnecessary notifications for prompting a start.

Means for Solving the Problems

[0006] To solve the above-mentioned problems and achieve the objective, the in-vehicle device according to the present invention has a controller that notifies the driver of the vehicle when the vehicle remains stationary after the traffic light changes from red to green and the vehicle does not start moving. The controller detects the presence or absence of the driver in the driver's seat from the interior image captured by the in-vehicle camera that captures the interior of the vehicle, and when the driver is not detected in the driver's seat, it switches to a stop mode in which the output of the notification is stopped. [Effects of the Invention]

[0007] According to the present invention, if the driver is not detected in the driver's seat, the output of notifications to the driver is stopped, thus eliminating unnecessary notifications prompting the driver to start moving. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of how an in-vehicle device can be installed. [Figure 2] Figure 2 shows a situation where unnecessary notifications are issued for ignoring a green light. [Figure 3] Figure 3 is an overview diagram illustrating the process for determining whether a green light has been ignored. [Figure 4] Figure 4 is a block diagram of the dashcam. [Figure 5] Figure 5 is a flowchart showing the processing procedure for determining whether a green light was ignored. [Figure 6] Figure 6 is a flowchart showing the processing procedure for driver detection. [Figure 7] Figure 7 is a flowchart showing the procedure for releasing the stop mode. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, the embodiments for implementing the in-vehicle device and control program according to the present application (hereinafter referred to as "embodiments"). However, these embodiments do not limit the in-vehicle device and control program according to the present application.

[0010] First, an example of mounting the in-vehicle device according to the embodiment will be described using Figure 1. Figure 1 is a diagram showing an example of mounting the in-vehicle device. As shown in Figure 1, the in-vehicle device according to the embodiment is a drive recorder 1 mounted on a vehicle 100.

[0011] The drive recorder 1 has an external camera that records the area in front of the vehicle 100 and an internal camera that records the interior of the vehicle 100. In this disclosure, the drive recorder 1 analyzes the external image captured by the external camera and determines the lighting status of the traffic lights captured by the external camera.

[0012] For example, if vehicle 100 is stopped at a red light and does not move for a certain period of time even after the traffic light changes from red to green, drive recorder 1 will notify driver D to start moving. In the following, the state in which vehicle 100 does not move for a certain period of time even after the traffic light changes from red to green will be referred to as "running a green light," and the judgment regarding running a green light will be referred to as "running a green light judgment."

[0013] In this disclosure, the drive recorder 1 has a function to stop unnecessary notifications prompting the driver to start moving when a green light is ignored. Here, using Figure 2, a specific example of a situation in which the drive recorder 1 stops prompting the driver to start moving will be explained. Figure 2 is a diagram showing a situation in which unnecessary notifications are issued when a green light is ignored.

[0014] As shown in Figure 2, assume that vehicle 100 is parked on the shoulder of the road and driver D is unloading cargo from vehicle 100. For example, in this situation, if traffic light 50 enters the imaging area of ​​the external camera, and the traffic light 50 changes from red to green, the conditions for ignoring a green light will be met.

[0015] That is, in the situation of FIG. 2, even though the driver D is absent, there is a possibility that a notification prompting a start will be issued, satisfying the determination condition for ignoring a green signal. Therefore, in the present disclosure, the drive recorder 1 detects the presence or absence of the driver D in the driver's seat of the vehicle 100, and when the driver D is absent, stops the notification of ignoring the green signal. Hereinafter, the mode of stopping the notification of ignoring the green signal will be described as the stop mode.

[0016] Thereby, the drive recorder 1 can stop an unnecessary notification prompting a start when ignoring a green signal.

[0017] Also, when the vehicle 100 is a commercial vehicle, the drive recorder 1 has a function of notifying an external device that manages the operation status of the vehicle 100 of the occurrence of ignoring a green signal when it is determined that the vehicle 100 has ignored a green signal. For example, the notification regarding the occurrence of ignoring a green signal includes the identification information of the vehicle 100, the occurrence time, images captured by an external camera or an in-vehicle camera, and the like.

[0018] For example, in the above stop mode, the drive recorder 1 stops notifying the external device of the occurrence of ignoring a green signal. Thereby, the drive recorder 1 can also reduce unnecessary communication regarding ignoring a green signal.

[0019] Next, a specific example of the process related to the determination of ignoring a green signal by the drive recorder 1 will be described using FIG. 3. FIG. 3 is a schematic explanatory diagram of the green signal ignoring determination process. First, the process regarding the external image captured by the external camera will be described.

[0020] First, the drive recorder 1 performs signal detection processing on the external image captured by the external camera. The signal detection processing is a process of detecting the traffic signal 50 shown in the external image. For example, the drive recorder 1 performs signal detection processing using a learning model (AI) that has learned the image of the traffic signal 50.

[0021] Next, the drive recorder 1 acquires signal information based on the results of the signal detection process. The signal information is information indicating the lighting status of the traffic light 50 detected by the signal detection process. In other words, the signal information is information regarding the current lighting color (red, blue, or yellow) of the traffic light 50.

[0022] Next, the drive recorder 1 determines whether a green light has been ignored based on the signal information. Specifically, the drive recorder 1 determines that a green light has been ignored if, while the vehicle 100 is stopped, the traffic light 50 changes from red to green, and the vehicle 100 remains stopped.

[0023] Then, drive recorder 1 makes a final judgment using the result of the green light violation detection. In parallel with the green light violation detection, drive recorder 1 also acquires interior images from the in-car camera and performs driver detection processing on the interior images.

[0024] The driver detection process is the process of detecting the presence or absence of driver D from the driver's seat of vehicle 100. For example, drive recorder 1 performs the driver detection process using a learning model that has learned images of people sitting in the driver's seat. For example, drive recorder 1 determines that driver D is in the driver's seat if driver D's face is detected, and determines that driver D is not in the driver's seat if driver D's face is not detected.

[0025] Next, drive recorder 1 calculates the number of times a driver was not detected based on the processing results of the driver detection process. The calculation of the number of times a driver was not detected is the process of calculating the number of times driver D was not in the driver's seat. For example, the count for calculating the number of times a driver was not detected is reset when driver D is detected in the driver's seat.

[0026] Drive recorder 1 turns off the undetected flag if the count for calculating the number of undetected events is below the threshold, and determines that driver D is not in the driver's seat if the count for calculating the number of undetected events exceeds the threshold.

[0027] In the final determination, if the processing result for the external image indicates a green light violation and the processing result for the internal image indicates an undetected flag is off, drive recorder 1 will notify driver D to start moving. In other words, drive recorder 1 will notify driver D to start moving if a green light violation occurs and driver D is in the driver's seat.

[0028] Furthermore, if the processing result for the in-vehicle image is set to "not detected" flag, drive recorder 1 switches to a stop mode, which stops notifying driver D to start driving. In other words, drive recorder 1 can suppress unnecessary notifications to driver D by switching to stop mode when the "not detected" flag is set to "on".

[0029] Thus, in the case of the drive recorder 1 according to this embodiment, if the driver D is not detected, the drive recorder 1 stops outputting a notification prompting the driver D to start moving.

[0030] Therefore, according to the drive recorder 1 of this embodiment, if driver D is not detected in the driver's seat, the output of notifications to driver D is stopped, thus eliminating unnecessary notifications prompting the driver to start moving. The condition for transitioning to stop mode is that the undetected state continues for a certain period of time, but it may also be the instantaneous detection of the undetected state.

[0031] Next, an example of the configuration of the drive recorder 1 according to the embodiment will be described using Figure 4. Figure 4 is a block diagram of the drive recorder 1. As shown in Figure 4, the drive recorder 1 comprises a communication unit 2, an external camera 3, an internal camera 4, a storage unit 5, and a controller 6.

[0032] Communication unit 2 is implemented by a network adapter or the like. Communication unit 2 is wirelessly connected to a network such as a mobile phone network or a C-V2X (Cellular Vehicle to Everything) communication network, and performs data communication with other devices via the network. For example, communication unit 2 performs data communication with a data center device that collects captured images.

[0033] The exterior camera 3 is mounted to capture the area in front of the vehicle 100. The interior camera 4 is mounted to capture the interior of the vehicle 100. In this disclosure, the interior camera 4 is mounted to capture the driver's seat. The exterior camera 3 and the interior camera 4 are mounted near the windshield, the rearview mirror, etc.

[0034] The memory unit 5 is implemented using memory devices such as ROM (Read Only Memory), RAM (Random Access Memory), and flash memory. The memory unit 5 stores multiple image recognition AI (Artificial Intelligence) systems.

[0035] The multiple image recognition AIs include an AI that detects traffic lights 50 and their illumination status from images outside the vehicle captured by the external camera 3, and an AI that detects the presence or absence of a driver D in the driver's seat from images inside the vehicle captured by the internal camera 4. Furthermore, these multiple image recognition AIs are, for example, Deep Neural Network (DNN) models trained using deep learning algorithms.

[0036] Controller 6 corresponds to a so-called processor. Controller 6 is implemented by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphical Processing Unit), etc. Controller 6 executes the control program according to the embodiment stored in the memory unit 5, using RAM as the working area. Controller 6 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0037] The controller 6 notifies the driver D of vehicle 100 if, while vehicle 100 is stopped, the traffic light 50 changes from red to blue, but vehicle 100 remains in a stationary state without starting.

[0038] The controller 6 analyzes the external images to calculate the vehicle speed 100 and determines whether the vehicle 100 is stationary or not. For example, the vehicle speed 100 can be calculated based on the amount of change (so-called optical flow) of the same feature points extracted from the time-series external images.

[0039] In this way, the controller 6 calculates the vehicle speed 100 from the external image, eliminating the need to connect a vehicle speed sensor or the like when installing the drive recorder 1, thus simplifying the installation of the drive recorder 1. Alternatively, the controller 6 may acquire vehicle speed information from a vehicle speed sensor.

[0040] Controller 6 uses the external image acquired from the external camera 3 to perform a traffic light violation detection process, and uses the internal image acquired from the internal camera 4 to perform a driver detection process. Finally, Controller 6 uses the results of the traffic light violation detection process and the driver detection process to make a final decision on whether or not to notify driver D to start moving.

[0041] If the controller 6 determines, based on the final judgment, that it should notify driver D to start moving, it will notify driver D to start moving via a speaker or display (not shown). Furthermore, if the controller 6 determines that it should notify driver D to start moving, it will notify external devices such as the center device via the communication unit 2 that a green light violation has occurred.

[0042] Next, the processing procedures performed by the drive recorder 1 according to this embodiment will be described using Figures 5 to 7. Figure 5 is a flowchart showing the processing procedure for determining whether a green light has been ignored. Figure 6 is a flowchart showing the processing procedure for driver detection. Figure 7 is a flowchart showing the processing procedure for canceling the stop mode. Note that the processing procedures shown below are performed by the controller 6.

[0043] First, the processing procedure for determining whether a green light has been ignored will be explained using Figure 5. As shown in Figure 5, first, the controller 6 acquires an external image from the external camera 3 (step S101), and then performs signal detection processing on the external image (step S102).

[0044] The signal detection process involves detecting the traffic light 50 in the external image of the vehicle, and is performed using image recognition AI. Subsequently, the controller 6 uses the processing result of the signal detection process to perform a green light disregard determination process (step S103).

[0045] The green light disregard determination process determines whether the vehicle 100 remains in a non-starting state after the traffic light 50 detected by the signal detection process changes from red to green.

[0046] The controller 6 can determine the lighting status of the traffic light 50 using image recognition AI, and further determine whether the vehicle 100 is in a state of not starting, for example, based on the optical flow obtained from the time-series external camera 3.

[0047] Next, the controller 6 determines whether the result of the green light violation detection is whether or not the driver violated the green light (step S104). If the controller 6 determines that the driver violated the green light (step S104; Yes), it determines whether or not the undetected flag is off (step S105). As mentioned above, the undetected flag is a flag that is turned on when driver D is not detected in the driver's seat for a certain period of time. The process of turning the undetected flag on and off will be described later using Figures 6 and 7.

[0048] If the controller 6 determines that the undetected flag is off (step S105; Yes), it notifies driver D to start moving (step S106) and terminates the process.

[0049] Furthermore, in the determination in step S104, if the controller 6 determines that the result of the green light violation determination is not a green light violation (step S104; No), it terminates the process. Also, in the determination in step S105, if the controller 6 determines that the undetected flag is on (step S105; No), it terminates the process.

[0050] Next, the processing procedure for driver detection will be explained using Figure 6. As shown in Figure 6, the controller 6 first acquires a vehicle image from the in-vehicle camera 4 (step S201), and then performs driver detection processing on the in-vehicle image (step S202).

[0051] The driver detection process detects the presence or absence of driver D in the driver's seat. For example, if driver D's face is detected from the in-car image, driver D in the driver's seat is detected. Subsequently, the controller 6 calculates the number of times the driver was not detected based on the processing result of the driver detection process (step S203).

[0052] The "undetected count" is a process that counts the frames in which driver D is absent. If driver D is not detected as a result of the driver detection process, drive recorder 1 adds "1" to the undetected count, and if driver D is detected, it resets the undetected count to "0".

[0053] Next, the controller 6 determines whether the number of undetected instances is greater than a threshold (step S204). If the controller 6 determines that the number of undetected instances is greater than a threshold (step S204; Yes), it turns on the undetected flag (step S205) and terminates the process. In other words, the controller 6 turns on the undetected flag if the undetected state, where driver D is not detected, continues for a certain period of time. When the controller 6 turns on the undetected flag, it switches to a stop mode in which it stops sending notifications to driver D prompting them to start moving.

[0054] Furthermore, if the controller 6 determines in step S204 that the number of undetected cases is below a threshold (step S204; No), it terminates the process.

[0055] Next, we will explain the procedure for canceling the stop mode using Figure 7. Note that the following procedure is executed repeatedly after the undetected flag is turned on in Figure 6.

[0056] As shown in Figure 7, when the controller 6 acquires an in-vehicle image from the in-vehicle camera 4 (step S301), it performs a driver detection process (step S302). As described above, the driver detection process is the process of detecting whether or not there is a driver D in the driver's seat.

[0057] Next, the controller 6 determines whether or not it has detected driver D as a result of the driver detection process (step S303). If the controller 6 determines that it has detected driver D (step S303; Yes), it determines whether or not the vehicle speed of vehicle 100 is greater than a threshold (step S304). The threshold here is, for example, 0 kilometers per hour, but can be set to any value.

[0058] If, in step S304, the controller 6 determines that the vehicle speed exceeds the threshold (step S304; Yes), it changes the undetected flag from on to off (step S305) and terminates the process. Also, if, in step S303, the controller 6 determines that driver D is not detected (step S303; No), it terminates the process. Also, if, in step S304, the controller 6 determines that the vehicle speed is below the threshold (step S304; No), it terminates the process.

[0059] As described above, in the determination in step S303, the drive recorder 1 detects driver D, and further, in the determination in step S304, if it determines that the vehicle speed exceeds a threshold, it changes the undetected flag from on to off and cancels the stop mode.

[0060] In other words, the drive recorder 1 deactivates the stop mode when vehicle 100 starts moving. This allows the drive recorder to appropriately prompt driver D to start moving if vehicle 100 then stops at a red light and the system determines that the vehicle has run a green light. Furthermore, with this configuration, it is also possible to stop notifications prompting the vehicle to start moving when vehicle 100 is a transporter's vehicle and the driver has returned to the driver's seat, for tasks such as completing a delivery or confirming the next delivery destination.

[0061] Furthermore, the controller 6 may, for example, change the undetected flag from on to off if it determines in step S303 that driver D has been detected. Alternatively, the controller 6 may omit the determination in step S303 and change the undetected flag from on to off if it determines in step S304 that the vehicle speed exceeds a threshold.

[0062] Furthermore, the controller 6 may determine when the vehicle 100 is starting based on the value of the G sensor mounted on the drive recorder 1. That is, the controller 6 may change the undetected flag from on to off when the G sensor detects an acceleration of a certain value or more in the direction of travel of the vehicle 100.

[0063] As described above, the drive recorder 1 (an example of an in-vehicle device) according to the embodiment has a controller 6 that notifies the driver D of the vehicle 100 when the vehicle 100 remains stationary after the traffic light 50 changes from red to blue.

[0064] The controller 6 detects the presence or absence of a driver in the driver's seat from the interior image captured by the in-vehicle camera 4, and switches to a stop mode that stops outputting notifications if a driver D is not detected in the driver's seat for a certain period of time.

[0065] Therefore, according to the drive recorder 1 of this embodiment, if driver D is not detected in the driver's seat, the output of notifications to driver D is stopped, thus eliminating unnecessary notifications prompting the vehicle to start moving.

[0066] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and equivalents. [Explanation of Symbols]

[0067] 1. Dashcam 2 Communications Department 3. Exterior car camera 4. In-car camera 5 Storage section 6 Controllers 50 Traffic lights 100 vehicles D Driver

Claims

1. The system has a controller that notifies the driver of a vehicle if, while the vehicle is stopped, the traffic light changes from red to green, and the vehicle remains in a state of not starting. The aforementioned controller, The presence or absence of the driver in the driver's seat is detected from the interior images captured by the in-vehicle camera that films the interior of the vehicle. In the driver's seat where the driver is not detected, the output of the notification is stopped. In-vehicle device.

2. The aforementioned controller, If the aforementioned undetected state continues for a certain period of time, the output of the notification will be stopped. The in-vehicle device according to claim 1.

3. The aforementioned controller, If the driver is detected in the driver's seat while the output of the aforementioned notification is stopped, the stop mode that stops the output of the aforementioned notification is released. The in-vehicle device according to claim 1.

4. The aforementioned controller, If the vehicle's speed exceeds a threshold while the output of the aforementioned notification is stopped, the stop mode that stops the output of the aforementioned notification is released. The in-vehicle device according to claim 1.

5. The aforementioned controller, If, while the output of the aforementioned notification is stopped, the driver is detected in the driver's seat and the vehicle's speed exceeds a threshold, the stop mode that stops the output of the aforementioned notification is released. The in-vehicle device according to claim 1.

6. The aforementioned controller, If the aforementioned non-starting state persists, the occurrence of the non-starting state is notified to an external device. While the output of the aforementioned notification is stopped, the notification to the external device of the occurrence of the non-starting state is also stopped. The in-vehicle device according to claim 1.

7. The aforementioned controller, Image recognition of the external image captured by the external camera that photographs the front of the vehicle detects that the state of the traffic light has changed from red to blue. The vehicle's speed is detected by image recognition of the external image. The in-vehicle device according to claim 4 or 5.

8. A control program executed by a controller that notifies the driver of a vehicle when the vehicle remains stationary after the traffic light changes from red to green, and the vehicle continues to remain stationary. The presence or absence of the driver in the driver's seat is detected from the interior images captured by the in-vehicle camera that films the interior of the vehicle. In the driver's seat where the driver is not detected, the output of the notification is stopped. Control program.

Images