In-vehicle devices and programs

Abstract

To provide more accurate driver assessments within intersections when entering an intersection on a yellow light. [Solution] The in-vehicle device according to the embodiment includes a controller. The controller detects the color of a traffic light based on image recognition processing of an image of the front of the vehicle captured by a camera. The controller also detects a change from a green light to a yellow light of the traffic light and determines that the vehicle has not yet entered the intersection, at which point it starts a driving diagnosis within the intersection. In the driving diagnosis, the controller also determines whether the driving is dangerous based on the vehicle's driving state, including either or both of the vehicle's acceleration and the direction of the driver's face.

Description

Technical Field

[0001] The disclosed embodiments relate to in-vehicle devices and programs.

Background Art

[0002] Conventionally, there has been known a technique for detecting the driving state of a vehicle based on information from various in-vehicle sensors including a camera, and performing a driving diagnosis of a driver based on the detection result. For example, such a technique includes detecting the color of a traffic signal from an image in front of the vehicle captured by a camera, and diagnosing the presence or absence of dangerous driving behaviors such as ignoring a red signal.

[0003] However, among such techniques, there are not many techniques for performing a driving diagnosis of a driver when entering an intersection on a yellow signal. Patent Document 1 discloses a technique for determining whether or not it is dangerous driving based on the temporal change in the distance between a vehicle and a traffic signal while including the case where the traffic signal is a yellow signal.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the above-described conventional technology still has room for further improvement in more accurately performing a driving diagnosis of a driver within an intersection when entering the intersection on a yellow signal.

[0006] While the general rule is to stop at a yellow light, if it is not possible to stop safely and the vehicle enters the intersection on a yellow light, the driver is required to proceed cautiously within the intersection. In other words, when entering an intersection on a yellow light, safe driving behavior is to proceed through the intersection without accelerating the vehicle, while thoroughly checking left and right. However, the conventional technology described above does not take into account such vehicle acceleration or left and right checks.

[0007] One embodiment of the invention has been made in view of the above, and aims to provide an in-vehicle device and program that can more accurately diagnose the driver's driving behavior within an intersection when entering the intersection on a yellow light. [Means for solving the problem]

[0008] An in-vehicle device according to one embodiment includes a controller. The controller detects the color of a traffic light based on image recognition processing of an image of the area in front of the vehicle captured by a camera. The controller also detects a change in the traffic light from green to yellow and, if it determines that the vehicle is about to enter an intersection, it starts a driving diagnosis. In the driving diagnosis, the controller also determines whether the driving is dangerous based on the vehicle's driving state, including either or both of the vehicle's acceleration and the direction of the driver's face. [Effects of the Invention]

[0009] According to one embodiment, when the controller detects a change from a green light to a yellow light, if the vehicle has not yet entered the intersection on the yellow light, it starts a driving diagnosis for entering the intersection on a yellow light. In this driving diagnosis, the controller determines whether the driving is dangerous based at least on the vehicle's acceleration and the direction of the driver's face. This makes it possible to judge dangerous driving based on safe driving behavior, such as passing through an intersection on a yellow light without accelerating the vehicle and while checking left and right sufficiently. Therefore, according to one embodiment, it is possible to perform a more accurate driving diagnosis of the driver within the intersection when entering the intersection on a yellow light. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a schematic diagram illustrating the operational diagnostic method according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of the driving diagnostic system according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of a drive recorder according to the embodiment. [Figure 4] Figure 4 shows an example of the configuration of a center device according to an embodiment. [Figure 5] Figure 5 is a flowchart showing the processing procedure performed by the drive recorder according to this embodiment. [Figure 6] Figure 6 is a flowchart showing the processing steps for calculating the degree of dangerous driving. [Figure 7] Figure 7 shows an example of the process for calculating the degree of dangerous driving. [Figure 8] Figure 8 is a flowchart showing the processing procedure for calculating the degree of dangerous driving related to the modified example. [Figure 9] Figure 9 shows an example of the conditions for adding to the dangerous driving score. [Figure 10] Figure 10 is a flowchart showing the processing procedure performed by the center device according to the embodiment. [Figure 11]FIG. 11 is a diagram showing an example of notification of a driving diagnosis report.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the in-vehicle device and program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

[0012] Also, hereinafter, it is assumed that the in-vehicle device according to the embodiment is the drive recorder 10 (see FIG. 1). It is assumed that the driving diagnosis method according to the embodiment is a driving diagnosis method executed by the controller 15 (see FIG. 3) of the drive recorder 10.

[0013] In addition, in the following description, expressions such as "specific", "predetermined", and "constant" may be read as "predetermined in advance".

[0014] First, an overview of the driving diagnosis method according to the embodiment will be described with reference to FIG. 1. FIG. 1 is a schematic explanatory diagram of the driving diagnosis method according to the embodiment.

[0015] In the driving diagnosis method according to the embodiment, the controller 15 of the drive recorder 10 detects the color of a traffic signal based on image recognition processing for an image of the front of the vehicle taken by a camera. In addition, when the controller 15 detects a change from a green signal to a yellow signal of the traffic signal and determines that the vehicle is before entering an intersection, the controller 15 starts a driving diagnosis. Further, in the above driving diagnosis, the controller 15 determines whether it is a dangerous driving based on the driving state of the vehicle including one or both of the acceleration of the vehicle and the direction of the driver's face.

[0016] Specifically, the drive recorder 10 is a video recording device mounted on a vehicle. The drive recorder 10 includes an external camera 12a, an in-vehicle camera 12b, and a G (acceleration) sensor 12c (see FIG. 3 for all).

[0017] The outside vehicle camera 12a is provided so as to be able to capture an outside vehicle image around the vehicle. The angle of view of the outside vehicle camera 12a is set to be able to capture at least the front of the vehicle. The inside vehicle camera 12b is provided so as to be able to capture an inside vehicle image inside the vehicle cabin. The angle of view of the inside vehicle camera 12b is set to be able to capture at least the face of the vehicle driver. The G sensor 12c measures the acceleration applied to the drive recorder 10.

[0018] The outside vehicle camera 12a, the inside vehicle camera 12b, and the G sensor 12c correspond to an example of an "in-vehicle sensor". Note that the "in-vehicle sensor" includes, in addition to these, for example, a GPS (Global Positioning System) sensor 12d, a vehicle speed sensor 5b, an accelerator sensor 5c, etc. (all are shown in FIG. 3).

[0019] The GPS sensor 12d measures the GPS position of the vehicle. The vehicle speed sensor 5b measures the speed (vehicle speed) of the vehicle. The accelerator sensor 5c measures the accelerator opening degree.

[0020] During vehicle startup, the drive recorder 10 can overwrite and record for a certain period of operation record data including the sensor data of these in-vehicle sensors (that is, outside vehicle image, inside vehicle image, acceleration, GPS position, vehicle speed, and accelerator opening degree) in the ring buffer memory. The certain period is, for example, 24 hours. The operation record data also includes, among other things, date and time information, etc. The operation record data is data indicating the driving state of the vehicle.

[0021] Also, in parallel with the recording of the operation record data, the drive recorder 10 executes image recognition processing on the outside vehicle image and the inside vehicle image using, for example, an AI (Artificial Intelligence) model for image recognition. The AI model is, for example, a DNN (Deep Neural Network) model learned using an algorithm of machine learning.

[0022] This AI model is pre-trained so that it can detect the type and position of each object shown in the video. In this embodiment, this AI model is pre-trained so that it can at least detect the traffic signal 500 in front of the vehicle, the color of the traffic signal 500, the crosswalk, the pedestrian W1, the bicycle B1, the other vehicle V1, and the direction of the driver's face as shown in FIG. 1.

[0023] The controller 15 of the drive recorder 10 can detect the change of the traffic signal 500 from green to yellow shown as the event E1 by performing image recognition processing on the out-of-vehicle video of the out-of-vehicle camera 12a using this AI model. Hereinafter, in some cases, the green signal may be simply referred to as "green" and the yellow signal may be simply referred to as "yellow".

[0024] When the controller 15 detects this event E1, it determines whether or not the driving diagnosis start condition for entering the intersection with a yellow signal is satisfied. Specifically, the controller 15 determines whether or not it is before entering the intersection based on the positional relationship between the tip position P1 of the vehicle with respect to the distances D1 and D2 (D1 < D2) from the intersection entrance at the time of detection of the event E1. If the controller 15 determines that such a positional relationship satisfies D1 ≤ P1 ≤ D2, it determines that it is before entering the intersection and starts a driving diagnosis for entering the intersection with a yellow signal (step S1). The distance D1 corresponds to an example of the "first distance". The distance D2 corresponds to an example of the "second distance".

[0025] The intersection entrance is the lower end position of the crosswalk that the vehicle first passes through when entering the intersection. The controller 15 calculates the distances D1 and the tip position P1 based on the image recognition processing result of the out-of-vehicle video of the out-of-vehicle camera 12a, and executes step S1 by comparing with a predetermined distance D2. Thereby, when the detection of the event E1 is before entering the intersection, the driving diagnosis for entering the intersection with a yellow signal can be appropriately started.

[0026] In the case where P1 < D1, the vehicle is close to the stop position and cannot stop safely. Therefore, the driver often has no choice but to enter the intersection as it is. In this case, since it is difficult to say that the driver entered the intersection voluntarily on the yellow signal, the controller 15 does not start the driving diagnosis within the intersection when entering the intersection on the yellow signal.

[0027] Also, in the case where D2 < P1, if the driver tries to enter the intersection, there is a high possibility that the red signal will be on during that time. In this case, it cannot be said that the driver entered the intersection on the yellow signal. Even if the driver entered the intersection, it is usually determined as signal violation. Therefore, the controller 15 does not start the driving diagnosis within the intersection when entering the intersection on the yellow signal.

[0028] On the other hand, when starting the driving diagnosis in step S1, the controller 15 determines whether it is a dangerous driving based on the driving state within the intersection indicated by the operation record data (step S2).

[0029] At this time, the controller 15 determines whether it is a dangerous driving based on, for example, the acceleration acquired from the G-sensor 12c or the movement of the direction of the driver's face detected by the image recognition process for the in-vehicle video of the in-vehicle camera 12b. For example, when it is determined that at least the driver has not accelerated the vehicle or checked left and right after detecting the event E1, the controller 15 determines that it is a dangerous driving.

[0030] Then, when the controller 15 determines that it is a dangerous driving in step S2, for example, it calculates the degree of dangerous driving according to the acceleration and the direction of the driver's face, and transmits the driving diagnosis information including such degree of dangerous driving and the operation record data to the center device 100 (step S3). The center device 100 is a device that collects the driving diagnosis information transmitted from the plurality of drive recorders 10.

[0031] Furthermore, the central device 100 performs a comprehensive driving diagnosis of each driver based on the driving diagnostic information collected from each drive recorder 10, and generates, for example, a driving diagnostic report for each driver. This makes it possible to perform a comprehensive driving diagnosis of each driver, including the results of the driving diagnosis within the intersection when entering the intersection on a yellow light. A specific example of the driving diagnostic report will be described later using Figure 11. A specific example of the method for calculating the degree of dangerous driving will be described later using Figures 6 to 9.

[0032] The controller 15 of the drive recorder 10 then determines whether or not the vehicle has passed through the intersection based on the image recognition processing results of the external video from the external camera 12a. If the controller 15 determines that the vehicle has passed through the intersection, it terminates the driving diagnosis within the intersection in the case of entering the intersection on a yellow light (step S4).

[0033] As described above, in the driving diagnostic method according to this embodiment, the controller 15 of the drive recorder 10 detects the color of the traffic light 500 based on image recognition processing of the image in front of the vehicle captured by the external camera 12a. The controller 15 also detects the change from a green light to a yellow light of the traffic light 500 and determines that the vehicle has not yet entered the intersection, at which point it starts the driving diagnostic. In addition, during the driving diagnostic, the controller 15 determines whether or not the driving is dangerous based on the vehicle's driving state, including either or both of the vehicle's acceleration and the direction of the driver's face.

[0034] Therefore, according to the driving diagnosis method of the embodiment, when the controller 15 detects a change from a green light to a yellow light at the traffic light 500, if the vehicle has not yet entered the intersection on the yellow light, it starts a driving diagnosis for entering the intersection on a yellow light. In this driving diagnosis, the controller 15 determines whether or not the driving is dangerous based at least on the vehicle's acceleration and the direction of the driver's face. This makes it possible to judge dangerous driving based on safe driving behavior, such as passing through an intersection on a yellow light while checking left and right sufficiently without accelerating the vehicle. Therefore, according to the drive recorder 10 of the embodiment, it is possible to perform a more accurate driving diagnosis of the driver within the intersection when entering the intersection on a yellow light.

[0035] The following describes in more detail an example of the configuration of a driving diagnostic system 1, including a drive recorder 10 to which the driving diagnostic method according to the above embodiment is applied.

[0036] Figure 2 shows an example of the configuration of the driving diagnostic system 1 according to the embodiment. As shown in Figure 2, the driving diagnostic system 1 includes drive recorders 10-1, 10-2, ..., 10-m (where m is a natural number of 3 or more), a center device 100, and application terminals 200-1, 200-2, ..., 200-n (where n is a natural number of 3 or more).

[0037] Each drive recorder 10, the central device 100, and each application terminal 200 are connected to each other via a network N, such as the internet, a mobile phone network, or a C-V2X (Cellular Vehicle to Everything) communication network.

[0038] The central device 100 is implemented, for example, as a private cloud. The central device 100 is managed, for example, by a business operator that operates the data center for the drive recorders 10 (e.g., an insurance company, a passenger transport company, a freight transport company, etc.). The central device 100 collects driving diagnostic information transmitted from each drive recorder 10.

[0039] Furthermore, the center device 100 performs a comprehensive driving diagnosis of the drivers of the vehicles equipped with each drive recorder 10 based on the collected driving diagnostic information. For example, the center device 100 generates and provides driving diagnostic reports for each driver as appropriate based on the collected driving diagnostic information.

[0040] The application terminal 200 is a computer capable of running application software (apps) specifically for the driving diagnostic system 1. The application terminal 200 may be, for example, a PC (Personal Computer) used by data center administrators or operators for managing driving diagnostic information. Alternatively, the application terminal 200 may be a mobile device such as a smartphone used by the driver of each vehicle.

[0041] The application terminal 200 used by data center administrators and operators is used, for example, via a data center application to perform various operations related to the management of driving diagnostic information. The application terminal 200 used by the drivers of each vehicle is used, for example, via a driver application to check driving diagnostic reports provided by the center device 100 and view video data associated with each piece of driving diagnostic information.

[0042] Next, an example of the configuration of the drive recorder 10 will be described. Figure 3 is a diagram showing an example of the configuration of the drive recorder 10 according to the embodiment. As shown in Figure 3, the drive recorder 10 has a communication unit 11, a sensor unit 12, an HMI unit 13, a storage unit 14, and a controller 15.

[0043] The communication unit 11 is implemented by a network adapter or the like. The communication unit 11 is wirelessly connected to the network N1 and transmits and receives information to and from the center device 100 via the network N1.

[0044] The sensor unit 12 is a group of various sensors mounted on the drive recorder 10. The sensor unit 12 includes an external camera 12a, an internal camera 12b, a G (accelerometer) sensor 12c, and a GPS sensor 12d. Since these sensors included in the sensor unit 12 have already been explained, their explanation here will be omitted.

[0045] In addition to the sensor unit 12, the drive recorder 10 is also connected to an external sensor unit 5, which is a group of various sensors mounted on the vehicle. The external sensor unit 5 includes an external camera 5a, a vehicle speed sensor 5b, and an accelerator sensor 5c. The external sensor unit 5 is connected to the drive recorder 10 via an in-vehicle network such as CAN (Controller Area Network).

[0046] The external camera 5a includes a rear camera that photographs the area behind the vehicle and a side camera that photographs the area to the side of the vehicle. The vehicle speed sensor 5b measures the vehicle speed, as previously mentioned. The accelerator sensor 5c measures the accelerator opening, as previously mentioned.

[0047] Furthermore, the drive recorder 10 is connected to an external device 7. The external device 7 includes ADAS (Advanced Driver-Assistance Systems) and various ECUs (Electronic Control Units). The external device 7 is connected to the drive recorder 10 so that they can communicate with each other via an in-vehicle network such as CAN. In addition, the external device 7 can, for example, use the recognition results from the image recognition processing of the drive recorder 10 to realize various advanced driver assistance functions and vehicle control functions.

[0048] The HMI unit 13 is a component that provides interface components for input and output to a user, such as a driver, who operates the drive recorder 10. The HMI unit 13 includes an input interface that accepts input operations from the user. The input interface is implemented, for example, by a touch panel. Alternatively, the input interface may be implemented by a microphone or the like. Furthermore, the input interface may be implemented by software components.

[0049] Furthermore, the HMI unit 13 includes an output interface for presenting visual and auditory information to the user. The output interface is implemented, for example, by a display or speaker. The HMI unit 13 may also provide the input interface and output interface to the user as an integrated unit, for example, by using a touch panel display.

[0050] The memory unit 14 is implemented using memory devices such as ROM (Read Only Memory), RAM (Random Access Memory), and flash memory. The memory unit 14 also includes a ring buffer memory. In the example shown in Figure 3, the memory unit 14 stores the operation record data DB (Database) 14a, event condition information 14b, image recognition model 14c, and driving diagnostic information 14d.

[0051] The operation record data DB14a is a database of operation record data recorded by the drive recorder 10. The event condition information 14b is information in which predetermined event conditions are set for detecting a specific event. In addition, the event condition information 14b is information associated with the action taken when a specific event corresponding to the event condition is detected.

[0052] For example, event condition information 14b defines the detection of the change from green to yellow of traffic light 500 in event E1 as an event, and the operations of steps S2 to S4 described above are associated with this event.

[0053] The image recognition model 14c corresponds to the AI ​​model for image recognition described above. After being loaded as an AI model into the controller 15, the image recognition model 14c functions as an image recognition AI that detects various objects in each frame when each frame of external or internal vehicle video is input to the controller 15.

[0054] The image recognition model 14c is configured to detect, for example, the traffic light 500, the color of the traffic light 500, the pedestrian crossing, pedestrians W1 and W2, bicycle B1, and other vehicles V1 that are visible in each frame when each frame of external video footage is input. The image recognition model 14c is also configured to detect, for example, the driver's face and facial feature point cloud that are visible in each frame when each frame of internal video footage is input.

[0055] The operational diagnostic information 14d is the information transmitted to the center device 100 in step S3 described above.

[0056] The controller 15 corresponds to a so-called processor. The controller 15 is implemented by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), etc. The controller 15 executes a program according to an embodiment not shown, stored in the memory unit 14, using RAM as the working area. The controller 15 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0057] The controller 15 performs information processing according to the processing procedures shown in Figures 5, 6, and 8. An explanation using Figures 5, 6, and 8 will be given later.

[0058] Next, an example of the configuration of the central device 100 will be described. Figure 4 is a diagram showing an example of the configuration of the central device 100 according to the embodiment. As shown in Figure 4, the central device 100 has a communication unit 101, a storage unit 102, and a controller 103.

[0059] The communication unit 101 is implemented by a network adapter or the like. The communication unit 101 is connected to the network N1 by wire or wireless connection and transmits and receives information to and from each drive recorder 10 and each application terminal 200 via the network N1.

[0060] The memory unit 102 is implemented by a storage device such as ROM, RAM, flash memory, or HDD (Hard Disk Drive). In the example shown in Figure 4, the memory unit 102 stores the operation diagnostic information DB 102a.

[0061] The driving diagnostic information DB102a is a database that stores driving diagnostic information, including the degree of dangerous driving and driving record data, collected from each drive recorder 10.

[0062] The controller 103 corresponds to a so-called processor. The controller 103 is implemented by a CPU, MPU, GPU, etc. The controller 103 executes a program according to an embodiment not shown, stored in the memory unit 102, using RAM as the working area. The controller 103 can also be implemented by an integrated circuit such as an ASIC or FPGA.

[0063] The controller 103 performs information processing according to the processing procedure shown in Figure 10. An explanation using Figure 10 will be given later.

[0064] Next, the processing steps performed by the drive recorder 10 will be explained using Figures 5 to 7. Figure 5 is a flowchart showing the processing steps performed by the drive recorder 10 according to this embodiment. Figure 6 is a flowchart showing the processing steps for the dangerous driving degree calculation process. Figure 7 is a diagram showing an example of the dangerous driving degree calculation process.

[0065] Although not shown in Figure 5, the controller 15 of the drive recorder 10 continuously acquires operational record data indicating the vehicle's driving status from the sensor unit 12 and the external sensor unit 5 after the vehicle has been started.

[0066] As shown in Figure 5, the controller 15 determines whether or not it has detected that the traffic light 500 ahead has changed from green to yellow (step S101). The controller 15 makes this determination based on the image recognition result of the external video captured by the external camera 12a or external camera 5a. If it has not detected the change (step S101, No), the controller 15 repeats step S101.

[0067] If detected (Step S101, Yes), the controller 15 determines whether the conditions for starting a driving diagnosis when the traffic light 500 changes from green to yellow are met (Step S102). That is, the controller 15 determines whether the front position P1 of the vehicle is between distances D1 and D2 from the intersection entrance (see Step S1 in Figure 1).

[0068] If the conditions for starting the driving diagnosis are not met (step S102, No), the controller 15 repeats the process from step S101. If the conditions for starting the driving diagnosis are met (step S102, Yes), the controller 15 sets the dangerous driving level to 0 (step S103).

[0069] The controller 15 then determines whether the acceleration is above a threshold (step S104). The acceleration is obtained from the G sensor 12c. Alternatively, the acceleration may be the change in vehicle speed obtained from the vehicle speed sensor 5b.

[0070] If the acceleration is above a threshold (step S104, Yes), the controller 15 determines whether the driver has not checked left or right (step S105). The controller 15 makes this determination based on the image recognition results of the in-vehicle video captured by the in-vehicle camera 12b.

[0071] For example, the controller 15 determines that the driver has not checked left and right if the movement of the driver's face direction, based on the image recognition results of the in-vehicle video, falls within a predetermined range. Note that the direction of the face may also be the driver's line of sight. This makes it possible to determine dangerous driving based on acceleration or the direction of the driver's face.

[0072] To explain the method for detecting the driver's face orientation or gaze, the controller 15 detects the driver's face orientation and movement based on the image recognition processing results of the in-vehicle video captured by the in-vehicle camera 12b, for example.

[0073] In this case, the controller 15 performs image recognition processing using the image recognition model 14c to detect the driver's face. The controller 15 also extracts a feature point cloud of the face within the detection frame where the face was detected.

[0074] The controller 15 then estimates the three-dimensional position and orientation of the driver's face as the face orientation based on the arrangement of the extracted feature point cloud. Specifically, the face position and orientation refer to the position and orientation of a particular part of the face (e.g., the area between the eyebrows).

[0075] Based on the arrangement of feature points, the controller 15 estimates the roll angle, tilt angle, and pan angle for three orthogonal axes with the center of the aforementioned specific part on the driver's midline as the origin, as the position and orientation of the face. Then, the controller 15 estimates the direction of the driver's line of sight as a three-dimensional vector corresponding to the estimated position and orientation of the face.

[0076] In this detection method (referred to as "the first detection method" for convenience), the driver's gaze is an estimate based on the direction of their face. However, since it is virtually impossible to change only the direction of the face without moving the eyes while driving a vehicle, the movement of the face can be considered identical to the movement of the gaze.

[0077] Furthermore, as an alternative method for detecting gaze, for example, an infrared camera may be included as an in-vehicle camera other than the in-vehicle camera 12b, and the controller 15 may detect the driver's gaze and its movement based on the infrared image captured by such an infrared camera.

[0078] In this detection method (referred to as the "second detection method" for convenience), the infrared camera is configured to include an infrared LED (Light Emitting Diode) that illuminates the driver's face with infrared light. The infrared camera is also mounted, for example, on the steering column, so that it can capture an infrared image of the driver's face illuminated by the infrared light.

[0079] In this case, the controller 15 estimates the direction of the driver's gaze based on the positional relationship between the driver's pupil in the infrared image and the infrared illumination reflection image generated on the eyeball. Because this second detection method is based on the positional relationship between the driver's actual pupil and the infrared illumination reflection image, it enables a more accurate estimation of the direction of the gaze compared to the first detection method. A configuration that enables the execution of the second detection method may be implemented, for example, for a Driver Monitoring System (DMS) in the ADAS described above.

[0080] Returning to the flowchart explanation, if the driver has not confirmed left or right (step S105, Yes), the controller 15 determines that it is dangerous driving (step S106). Then, the controller 15 executes the dangerous driving degree calculation process (step S107).

[0081] On the other hand, if the acceleration is below the threshold (step S104, No), or if the driver checks left and right (step S105, No), the controller 15 proceeds to step S111 without determining that it is dangerous driving. Note that at least one of steps S104 or S105 may be executed.

[0082] In the dangerous driving degree calculation process, the controller 15 calculates the dangerous driving degree from the acceleration and face orientation, as shown in Figure 6 (step S201). Specifically, as shown in Figure 7, the controller 15 calculates the dangerous driving degree as "1" (somewhat dangerous) if, for example, the acceleration is greater than X1 and the face orientation (movement) is less than Y1+y1+y2.

[0083] Furthermore, the controller 15 calculates the degree of dangerous driving as "2" (dangerous) if, for example, the acceleration is greater than X1+x1 and the direction of the face is less than Y1+y1. Also, the controller 15 calculates the degree of dangerous driving as "3" (very dangerous) if, for example, the acceleration is greater than X1+x1+x2 and the direction of the face is less than Y1. Note that the relationship between each threshold shown in Figure 7 is X1<(X1+x1)<(X1+x1+x2) or (Y1+y1+y2)>(Y1+y1)>Y1.

[0084] In other words, the controller 15 calculates the degree of dangerous driving such that the greater the acceleration and the smaller the head movement, the higher the degree of dangerous driving. This makes it possible to judge dangerous driving based on safe driving behavior, such as entering an intersection on a yellow light without accelerating the vehicle and passing through the intersection while checking left and right sufficiently. Note that the example shown in Figure 7 is just one example and does not limit the method of calculating the degree of dangerous driving. For example, although each applicable condition shown in Figure 7 is an AND (&) condition, an OR condition may also be used.

[0085] Returning to the explanation of Figure 5, after the dangerous driving degree calculation process is executed, the controller 15 determines whether or not there has been a change in the dangerous driving degree (step S108). If there has been a change (step S108, Yes), the controller 15 issues a warning to the driver via the HMI unit 13 (step S109). "Warning" may be read as "attention alert".

[0086] In step S109, the controller 15 provides notifications via voice output or display output, for example, based on acceleration or head orientation, such as "You are accelerating too much" or "Please check left and right more carefully." This allows the driver to easily understand their own driving shortcomings. The controller 15 then transmits driving diagnostic information, including the calculated dangerous driving degree and driving record data, to the center device 100 (step S110).

[0087] If there is no change in the degree of dangerous driving (step S108, No), the controller 15 proceeds to step S111. Then, the controller 15 determines whether or not the driving diagnosis completion conditions are met (step S111).

[0088] In other words, the controller 15 determines whether the vehicle has passed the intersection exit (see step S4 in Figure 1). If the driving diagnosis termination conditions are not met (step S111, No), the controller 15 repeats the process from step S104. If the driving diagnosis termination conditions are met (step S111, Yes), the controller 15 determines whether the system has terminated (step S112). The controller 15 determines that the system has terminated, for example, if the ignition switch is turned off.

[0089] If the system is not shutting down (step S112, No), controller 52 repeats the process from step S101. If the system is shutting down (step S112, Yes), controller 15 terminates the process.

[0090] Next, a modified version of the dangerous driving degree calculation process will be explained using Figures 8 and 9. Figure 8 is a flowchart showing the processing steps of the dangerous driving degree calculation process related to the modified version. Figure 9 is a diagram showing an example of the conditions for adding dangerous driving degrees. Note that α1 to α5 shown in Figure 8 are all arbitrary constants.

[0091] In the modified version of the dangerous driving degree calculation process, the controller 15 calculates the dangerous driving degree from the acceleration and face orientation, as shown in Figure 8 (step S301). Step S301 is the same as step S201, which was explained using Figures 6 and 7.

[0092] Next, the controller 15 determines whether there is a pedestrian W1, a bicycle B1, or another vehicle V1 near the crosswalk (step S302). If there is a pedestrian W1, a bicycle B1, or another vehicle V1 near the crosswalk (step S302, Yes), the controller 15 adds α1 to the dangerous driving score (step S303).

[0093] The controller 15 then determines whether there is movement of pedestrian W1, bicycle B1, or other vehicle V1 near the crosswalk (step S304). If there is movement of pedestrian W1, bicycle B1, or other vehicle V1 (step S304, Yes), the controller 15 adds α2 to the dangerous driving score (step S305). This makes it possible to determine dangerous driving based on the presence or movement of pedestrian W1, bicycle B1, or other vehicle V1 near the crosswalk.

[0094] If there are no pedestrians W1, bicycles B1, or other vehicles V1 near the crosswalk (step S302, No), or if there is no movement of pedestrians W1, bicycles B1, or other vehicles V1 (step S304, No), the controller 15 proceeds to step S306. The controller 15 makes the determination in step S302 or step S304 based on the image recognition results of the external video captured by the external camera 12a or external camera 5a.

[0095] Next, in step S306, the controller 15 determines whether the elapsed time since the traffic light 500 turned yellow is greater than or equal to a threshold. If it is greater than or equal to the threshold (step S306, Yes), the controller 15 adds α3 to the dangerous driving degree (step S307). This allows dangerous driving to be judged based on the elapsed time since the traffic light 500 turned yellow. If it is less than the threshold (step S306, No), the controller 15 proceeds to step S308.

[0096] Next, in step S308, the controller 15 determines whether the accelerator opening has increased within a predetermined time since the traffic light 500 turned yellow (step S308). That is, the controller 15 determines whether the driver pressed the accelerator further after the traffic light 500 turned yellow. The accelerator opening is obtained from the accelerator sensor 5c.

[0097] If the accelerator opening increases within a predetermined time (step S308, Yes), the controller 15 adds α4 to the dangerous driving degree (step S309). This allows the controller to determine if the driver pressed the accelerator further after the traffic light 500 turned yellow. If the accelerator opening does not increase within a predetermined time (step S308, No), the controller 15 proceeds to step S310.

[0098] Next, in step S310, the controller 15 determines whether the external environment meets the conditions for adding to the dangerous driving degree. If the external environment meets the conditions for adding to the dangerous driving degree (step S310, Yes), the controller 15 adds α5 to the dangerous driving degree (step S311) and terminates the dangerous driving degree calculation process related to the modified example. If the conditions for adding to the dangerous driving degree are not met (step S310, No), the controller 15 terminates the dangerous driving degree calculation process related to the modified example.

[0099] As shown in Figure 9, the conditions for adding to the dangerous driving score include adverse weather conditions such as rain or cloudy skies, nighttime hours when blind spots are more likely to occur, and commuting hours when there are many pedestrians. Furthermore, the conditions for adding to the dangerous driving score include twilight hours, which are considered to be prone to accidents, and the current location being near a school with many children. By adding to the dangerous driving score according to various external environments that are assumed to be high-risk in this way, it is possible to contribute to more accurate driving diagnosis.

[0100] Next, we will explain the processing procedure performed by the center device 100. Figure 10 is a flowchart showing the processing procedure performed by the center device 100 according to this embodiment.

[0101] As shown in Figure 10, the center device 100 determines whether the controller 103 has received driving diagnostic information from the vehicle side (step S401). If it has not received the information (step S401, No), the controller 103 repeats step S401. If it has received the information (step S401, Yes), the controller 103 stores the received driving diagnostic information in the driving diagnostic information DB 102a (step S402). The controller 103 continuously repeats steps S401 and S402.

[0102] Then, at any time, the controller 103 generates a driving diagnostic report for each driver based on the driving diagnostic information stored in the driving diagnostic information DB 102a (step S403). The controller 103 then provides the generated driving diagnostic report to the drive recorder 10 and the application terminal 200 (step S404).

[0103] Figure 11 shows an example of a driving diagnostic report notification. Although Figure 11 shows an example of a notification displaying the driving diagnostic report screen M1 to the HMI unit 13 of the drive recorder 10, a similar notification may be sent to the application terminal 200.

[0104] As shown in Figure 11, the controller 103 generates a driving diagnostic report screen M1 that includes a list of the date and time, location, event, operation, degree of dangerous driving, etc., when, for example, a vehicle enters an intersection on a yellow light and the driving condition within the intersection is determined to be dangerous driving. In this case, the event is the change from green to yellow of traffic light 500. In this case, the operation is going straight or turning right or left.

[0105] The controller 103 also generates a driving diagnostic report screen M1 which includes the driver's safe driving score and improvement comments. The safe driving score is calculated, for example, by deducting points from a maximum score of 100 based on the cumulative number of dangerous driving incidents and the degree of each dangerous driving incident. Improvement comments are generated, for example, by a generation AI that has been pre-trained to generate improvement comments using driving diagnostic information stored in the driving diagnostic information DB 102a as input. Alternatively, improvement comments may be generated without using a generation AI, for example, by using a pre-prepared template statement.

[0106] As shown in Figure 11, such improvement comments can, for example, point out the driver's quirks, thereby providing the driver with specific areas for improvement.

[0107] As described above, the drive recorder 10 according to the embodiment (corresponding to an example of an "in-vehicle device") includes a controller 15. The controller 15 detects the color of the traffic light 500 based on image recognition processing of the image in front of the vehicle captured by the external camera 12a. The controller 15 also detects the change from a green light to a yellow light of the traffic light 500 and, if it determines that the vehicle has not yet entered the intersection, it starts a driving diagnosis. In the driving diagnosis, the controller 15 also determines whether or not the driving is dangerous based on the vehicle's driving state, including either or both of the vehicle's acceleration and the direction of the driver's face.

[0108] Therefore, according to the drive recorder 10 according to this embodiment, when the controller 15 detects a change from a green light to a yellow light at the traffic light 500, if the vehicle has not yet entered the intersection on the yellow light, it starts a driving diagnosis for entering the intersection on a yellow light. In this driving diagnosis, the controller 15 determines whether or not the driving is dangerous based at least on the vehicle's acceleration and the direction of the driver's face. This makes it possible to judge dangerous driving based on safe driving behavior, such as passing through the intersection on a yellow light while checking left and right sufficiently without accelerating the vehicle. Therefore, according to the drive recorder 10 according to this embodiment, it is possible to perform a more accurate driving diagnosis of the driver within the intersection when entering the intersection on a yellow light.

[0109] In the embodiment described above, the controller 15 determines whether the vehicle has entered the intersection before the event E1 is detected, based on the positional relationship of the vehicle's front end position P1 with respect to distances D1 and D2 from the intersection entrance. If this positional relationship satisfies D1 ≤ P1 ≤ D2, the controller 15 starts a driving diagnosis for entering the intersection on a yellow light (see step S1 in Figure 1).

[0110] As a variation of this case, for example, if there is a vehicle behind and the distance between it and the vehicle is short, there is a risk of a rear-end collision if the vehicle does not accelerate or decelerate in the intersection. In this case, the driving diagnosis for entering the intersection on a yellow light may not be initiated. Alternatively, even if the driving diagnosis is initiated, at least acceleration may not be subject to penalty. The presence of a vehicle behind and the distance between them can be detected based on the image recognition processing results of the rear view captured by the rear camera included in the external camera 5a.

[0111] Furthermore, the distance D1 described in the above-described embodiment may be converted into driving time. Also, although the above-described embodiment described the driving diagnostic information as mainly relating to dangerous driving, the driving diagnostic information may also relate to safe driving. That is, if the driving is not judged as dangerous driving even once from the start to the end of the driving diagnostic, the driving diagnostic information may be transmitted to the center device 100 as information diagnosed as safe driving. In this case, the center device 100 may treat such information as a target for adding points to the safe driving score. Furthermore, if the safe driving score is higher than a threshold, the center device 100 may generate a driving diagnostic report that includes praise comments instead of improvement comments.

[0112] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0113] 1. Driving diagnostic system 5. External sensor section 7 External device 10. Dashcam 11 Communications Department 12 Sensor section 13 HMI section 14 Storage section 15 Controllers 100 Center device 101 Communications Department 102 Storage section 103 Controller

Claims

1. It is equipped with a controller that detects the color of traffic lights based on image recognition processing of images of the front of the vehicle captured by a camera, The aforementioned controller, When the system detects a change from a green light to a yellow light for the aforementioned traffic signal and determines that the vehicle has not yet entered the intersection, it initiates a driving diagnosis. In the aforementioned driving diagnosis, a determination is made as to whether or not the driving is dangerous based on the driving state of the vehicle, including either or both of the vehicle's acceleration and the direction of the driver's face. In-vehicle device.

2. The aforementioned controller, If the front end of the vehicle is located between a first distance from the entrance to the intersection, which is close to the entrance, and a second distance from the entrance that is longer than the first distance, it is determined that the vehicle has not yet entered the intersection. The in-vehicle device according to claim 1.

3. The aforementioned controller, If the acceleration is above a threshold, or if the driver has not checked left and right based on the direction of their face, it is determined that the driving is dangerous. The in-vehicle device according to claim 1.

4. The aforementioned controller, If it is determined that the driving is dangerous, the degree of dangerous driving is calculated based on the acceleration or the direction of the face. The in-vehicle device according to claim 3.

5. The aforementioned controller, The driver is given a warning according to the degree of dangerous driving. The in-vehicle device according to claim 4.

6. The aforementioned controller, When a pedestrian, bicycle, or other vehicle is detected at the intersection based on the image recognition process, the degree of dangerous driving is increased. The in-vehicle device according to claim 4.

7. The aforementioned controller, If the elapsed time since the traffic light changed to yellow exceeds a threshold, or if the accelerator opening increases within a predetermined time after the traffic light changed to yellow, the degree of dangerous driving is increased. The in-vehicle device according to claim 4.

8. The aforementioned controller, If external environmental factors, including weather, time of day, or current location, meet the specified conditions for increasing the degree of dangerous driving, the degree of dangerous driving will be increased. The in-vehicle device according to claim 4.

9. The aforementioned controller, The system transmits operational diagnostic information, including the degree of dangerous driving and data indicating the driving state associated with the degree of dangerous driving, to the center device. The aforementioned center device is Based on the aforementioned driving diagnostic information, a driving diagnostic report is generated for each driver. The driver provides the aforementioned operational diagnostic report to the terminal device used by the driver. The in-vehicle device according to any one of claims 4 to 8.

10. Based on image recognition processing of the image of the front of the vehicle captured by the camera, the color of the traffic light is detected. When the system detects a change from a green light to a yellow light for the aforementioned traffic signal and determines that the vehicle has not yet entered the intersection, it initiates a driving diagnosis. In the aforementioned driving diagnosis, a determination is made as to whether or not the driving is dangerous based on the driving state of the vehicle, including either or both of the vehicle's acceleration and the direction of the driver's face. The program executed by the controller.

Images