Driving assistance device and driving assistance method

Abstract

The present invention evaluates the driving skill of a driver on the basis of the order of safety confirmation actions of the driver, and provides driving assistance according to the evaluated driving skill. A driving assistance device for assisting the driving of a driver comprises: a driver information acquisition unit for acquiring driver information including line-of-sight information about the driver; a vehicle information acquisition unit for acquiring vehicle information about a vehicle being driven by the driver; a transition state generation unit for generating transition states of actions related to the driving of the driver by using the driver information and the vehicle information; a driving score calculation unit for calculating a driving score for an action of the driver on the basis of a correspondence table of scores and the transition states of actions generated by the transition state generation unit; a calculation reason creation unit for creating a calculation reason for the driving score calculated by the driving score calculation unit by using the transition states of the actions; and an output unit for outputting the driving score and the calculation reason for the driving score.

Description

Driving Support Device and Driving Support Method

[0001] The present invention relates to a driving support device and a driving support method that evaluate a driver's driving skill and support driving according to the evaluated driving skill.

[0002] In recent years, vehicles equipped with ADAS (Advanced Driving Assistance System) (hereinafter referred to as "ADAS vehicles") have been rapidly spreading. Also, as a function of ADAS, a system that changes the mode of driving support according to the driver's safety confirmation behavior has been proposed.

[0003] For example, the abstract of Patent Document 1 states that "The safety confirmation diagnosis system of the present invention includes a camera (2) for photographing the driver's face, a face orientation detection unit (3) for detecting the orientation of the driver's face based on the image data of the driver's face, a driving scene detection unit (9) for detecting the driving scene of the vehicle, an evaluation section division unit (10) for dividing the driving scene into evaluation sections with different evaluation criteria, and a safety confirmation behavior evaluation unit (11) for evaluating the driver's safety confirmation behavior based on the orientation of the driver's face and the evaluation section of the vehicle's driving scene."

[0004] Also, in paragraph 0057 of the same document, it is stated that "Even when executing autonomous driving such as auto cruise or lane keep assist, since the responsibility for safe operation lies with the driver, the driver always needs to perform safety confirmation. Therefore, in the second embodiment as well, the safety confirmation behavior is evaluated for each evaluation section of each driving scene, and when it is determined that the driver's safety confirmation behavior is not being performed appropriately, advice is presented to perform the safety confirmation behavior, or the safety confirmation direction is presented. Also, in the second embodiment, when it is repeatedly determined that the driver's safety confirmation behavior is not being performed appropriately, the operation mode of the autonomous driving device is changed, for example, the vehicle speed is controlled to be below the set value, or the vehicle lane change is made impossible." Thus, Patent Document 1 discloses a safety confirmation diagnosis system that prompts the driver to perform safety confirmation or restricts the mode of autonomous driving when it is determined that the safety confirmation behavior is not being performed appropriately.

[0005] Japanese Unexamined Patent Application Publication No. 2017 - 151694

[0006] However, the safety confirmation diagnostic system described in Patent Document 1 does not evaluate safety confirmation behavior based on the time-series changes in driver behavior. Therefore, in the safety confirmation diagnostic system of Patent Document 1, if there is inappropriate behavior in any of the evaluation intervals, the safety confirmation behavior as a whole may be judged as inappropriate, regardless of the importance of that evaluation interval, making it difficult to perform a flexible evaluation of the order of the driver's safety confirmation behaviors.

[0007] Therefore, the present invention aims to provide a driver assistance device and a driver assistance method that evaluate the driver's driving skills based on the sequence of the driver's safety confirmation actions, and that provide driver assistance according to the evaluated driving skills.

[0008] To solve the above problems, one driver assistance device is a driver assistance device that assists a driver in driving, and comprises: a driver information acquisition unit that acquires driver information including the driver's gaze information; a vehicle information acquisition unit that acquires vehicle information of the vehicle being driven by the driver; a transition state generation unit that generates transition states of the driver's actions using the driver information and the vehicle information; a driving score calculation unit that calculates a driving score for the driver's actions based on a correspondence table of the action transition states and scores generated by the transition state generation unit; a calculation reason creation unit that creates a reason for calculating the driving score calculated by the driving score calculation unit using the action transition states; and an output unit that outputs the driving score and the reason for calculating the driving score.

[0009] The driver assistance device and driver assistance method of the present invention make it possible to evaluate the driver's driving skills based on the sequence of the driver's safety confirmation actions, and to provide driver assistance according to the evaluated driving skills.

[0010] Hardware configuration diagram of the driver assistance device of Example 1. Functional block diagram of the driver assistance device of Example 1. Plan view illustrating the vehicle's driving environment. Flowchart explaining an example of the operation of the driver assistance device of Example 1. Diagram explaining the state transition score of Example 1. Flowchart explaining the score calculation in Example 1. Functional block diagram of the driver assistance device of Example 2. Flowchart explaining an example of the operation of the driver assistance device of Example 2. Action transition diagram when entanglement check is performed in Example 2. Action transition diagram when entanglement check is not performed in Example 2. Flowchart explaining the score calculation in Example 2.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0012] First, using Figures 1A to 5, we will describe the driver assistance device 1 according to Embodiment 1 of the present invention, which is installed in an ADAS vehicle.

[0013] <ADAS Vehicles> First, let's explain ADAS vehicles. ADAS vehicles are vehicles equipped with an Advanced Driver Assistance System (ADAS), and have external sensors, vehicle sensors, and an HMI (Human Machine Interface) as components for realizing driver assistance control.

[0014] External sensors are sensors that can recognize the surrounding environment of a vehicle, and include, for example, monocular cameras, stereo cameras, laser radar, millimeter-wave radar, ultrasonic sensors, and infrared sensors.

[0015] Vehicle sensors are sensors capable of acquiring vehicle position information, longitudinal acceleration, yaw rate, etc. Examples include wheel speed sensors, acceleration sensors, angular velocity sensors, and Global Navigation Satellite System (GNSS) terminals.

[0016] HMI (Human-Machine Interface) is a device for interacting with the driver, and includes eye trackers that acquire the driver's gaze direction, in-car cameras, touch-type displays that the driver can operate, and voice devices that output warnings.

[0017] <Driving Assistance Device 1> Figure 1A is a hardware configuration diagram of the driving assistance device 1 of this embodiment. As illustrated here, the driving assistance device 1 is a computer system in which the hardware, including an input interface for receiving information from external sensors, an output interface for outputting information to the HMI, a storage device such as a semiconductor memory for storing information, and a computing device such as a CPU (Central Processing Unit) for processing information, are interconnected via an internal bus. Note that some or all of this hardware may be replaced with dedicated devices, general-purpose machine learning machines, DSPs (Digital Signal Processors), FPGAs (Field-Programmable Gate Arrays), GPUs (Graphics Processing Units), PLDs (Programmable Logic Devices), etc.

[0018] Furthermore, Figure 1B is a functional block diagram of the driver assistance device 1 of this embodiment, which is realized by the hardware of Figure 1A working together to execute a predetermined driver assistance program. As shown here, the driver assistance device 1 includes the following functional units: a driver information acquisition unit 11, a vehicle information acquisition unit 12, a transition state generation unit 13, a driving score calculation unit 14, a calculation reason creation unit 15, and an output unit 16.

[0019] <Example of Operation of Driving Assistance Device 1> Figure 2 is a plan view illustrating the driving environment of an ADAS vehicle (hereinafter referred to as "vehicle 21") equipped with the driving assistance device 1 of this embodiment, showing a situation in which vehicle 21 is about to turn left at an intersection. Figure 3 is a flowchart explaining an example of operation of the driving assistance device 1 while vehicle 21 is in motion. The following explanation will proceed sequentially, based on the driving environment shown in Figure 2, and each step in Figure 3.

[0020] <<Step S31>> First, in step S31, the driver information acquisition unit 11 acquires driver information from the HMI or the like. This driver information includes information such as the driver's line of sight and head rotation angle. By utilizing this information, the driver information acquisition unit 11 can determine what the driver is looking at. If the object to be looked at has already been calculated by another device, that information may be acquired as driver information.

[0021] Eye-line information is generally represented as a single point (viewpoint), but the surrounding area is often also visible. Therefore, eye-line information can be represented as a field of view, formed from the viewpoint and the surrounding area. Eye-line information is information indicating the direction in which the driver is actually looking with their eyes. This eye-line information can be acquired directly by an eye tracker worn by the driver, or it can be estimated using images captured by an in-car camera. Eye-line information can also be estimated from the driver's gaze information and head turn angle.

[0022] <<Step S32>> In step S32, the vehicle information acquisition unit 12 acquires vehicle information from vehicle sensors and external sensors. This vehicle information includes vehicle motion information such as vehicle speed and steering angle, position information acquired from GNSS, and various external information obtained from external sensors. Generally, vehicle motion information that can be acquired from vehicle sensors includes, but is not limited to, vehicle speed, longitudinal acceleration, lateral acceleration, yaw rate, steering angle, and steering angular velocity. Furthermore, longitudinal acceleration can be calculated from the time derivative of vehicle speed, and lateral acceleration can be calculated from the relationship between yaw rate and velocity, so it is not always necessary to directly obtain physical quantities.

[0023] <<Step S33>> In step S33, the vehicle information acquisition unit 12 estimates the driving state of the vehicle 21 based on the vehicle information acquired in step S32. Specifically, the driving state estimated here is the type of driving scene. In the driving environment shown in Figure 2, the driving state is estimated to be a "left turn at an intersection scene". The driving state may also be estimated by dividing it into more detailed driving scenes. For example, the "left turn at an intersection scene" in Figure 2 may be divided into three scenes: before entering the intersection 22, inside the intersection 23, and exiting the intersection 24.

[0024] <<Step S34>> In step S34, the vehicle information acquisition unit 12 determines whether the driving state has changed. If the driving state has changed, the process proceeds to step S35; otherwise, the process returns to step S31. Here, a change in the driving state would be, in the example shown in Figure 2, when the "left turn at an intersection scene" ends and the process transitions to the "straight driving scene".

[0025] <<Step S35>> In step S35, the transition state generation unit 13 etc. generates a state transition score. Here, the state transition score represents the quality of the action transitions that the driver can take in various driving scenarios, expressed as a score of magnitude.

[0026] Figure 4 shows an example of a state transition score table referenced in this step. The score table shown here is a set of information that represents the correspondence between the sequence of actions a driver can take in a "left turn at an intersection scene" and the state transition scores for those sequences of actions.

[0027] For example, if a driver's sequence of actions is a combination of past state "paying attention ahead" and current state "paying attention ahead," the driver is only focusing on what is in front of them (in other words, not paying attention to anything else), so this sequence of actions will receive a low rating (score 0). On the other hand, if the sequence of actions is a combination of past state "paying attention ahead" and current state "checking the rearview mirror," the driver is paying attention to both what is in front and what is behind, so this sequence of actions will receive a high rating (score 1). Similarly, if the sequence of actions is a combination of past state "checking for blind spots" and current state "turning left," the driver is turning left after checking for blind spots, so this sequence of actions will receive a higher rating (score 2).

[0028] Needless to say, the states (actions) in the state transition score table in Figure 4 are changed according to the conditions of the road. For example, at the intersection in Figure 2, since there is a traffic light 25 ahead, the action of "checking the traffic light" may be added as a past state or present state.

[0029] Note that in the score table in Figure 4, state transition scores were generated based on combinations of two consecutive states, but state transition scores may also be generated based on combinations of three or more consecutive states. In that case, state transition scores may also be generated according to the order of the three or more states.

[0030] Figure 5 is a flowchart illustrating the details of step S35.

[0031] First, in step S35a, the driver information acquisition unit 11 acquires driver information in the same manner as in step S31.

[0032] In step S35b, the vehicle information acquisition unit 12 acquires vehicle information in the same manner as in step S32.

[0033] In step S35c, the transition state generation unit 13 estimates the driver's actions based on the driver information and vehicle information acquired in steps S35a and S35b.

[0034] In step S35d, the transition state generation unit 13 determines whether the driver's behavior has changed. If the driver's behavior has changed, the process proceeds to step S35e; otherwise, it returns to step S35a.

[0035] In step S35e, the driving score calculation unit 14 adds state transition scores based on the current driver behavior, while referring to the state transition score table illustrated in Figure 4.

[0036] In step S35f, the driving score calculation unit 14 updates the state transition score table based on the actions the driver has taken up to that point. For example, after checking the rearview mirror once, the scores in the row where the current state in Figure 4 is checking the rearview mirror may all be set to 0. By repeating these steps, the score for each driving state is calculated, and the driver's driving skill is determined.

[0037] Furthermore, the method for calculating the score in step S35 may be further refined. For example, in the safety check under the environment shown in Figure 2, performing a check for blind spots inside the intersection 23, when it should have been performed before entering the intersection 22, is not considered appropriate action. Therefore, a high evaluation (score 1) may be calculated only when a safe action is performed within a predetermined area, and a low evaluation (score 0) may be calculated if the same action is performed at an inappropriate time. To determine whether the vehicle is traveling within a predetermined area, map information and vehicle position information, such as those installed in a navigation system, can be used.

[0038] Alternatively, the state transition score may be calculated separately for visual skills and operational skills. Examples of visual skills include the actions shown at the top of Figure 4. For operational skills, the ability to perform actions such as deceleration, acceleration, and steering in a sequential manner is evaluated.

[0039] <<Step S36>> In step S36, the driving score calculation unit 14 calculates a driving score based on the driver's behavior history. For example, as shown in Figure 2, when vehicle 21 turns left at an intersection, the driver performs the left turn while keeping an eye on the road ahead, checking the rearview mirror, checking the side mirrors, and checking for blind spots. Therefore, the driving score calculation unit 14 obtains a driving score by summing up the individual state transition scores generated by the transition state generation unit 13 in step S35.

[0040] <<Step S37>> In step S37, the calculation reason creation unit 15 stores time-series data of the driver's driving behavior. At this time, the calculation reason creation unit 15 may not only store time-series data of actual driving behavior, but may also estimate and store exemplary behavioral transitions that would result in a higher score. By presenting these exemplary behavioral transitions to the driver, the driver can reflect on their driving afterward and use this to improve their driving skills.

[0041] <<Step S38>>In step S38, the output unit 16 outputs a warning to the driver according to the results of steps S36 and S37. As methods for outputting warnings, there are various methods such as warning sounds, voice guidance, and drawing on a liquid crystal panel or HUD (Head Up Display). In this embodiment, the method is not limited.

[0042] Also, the degree of warning may be changed according to the magnitude of the driving score. It is also possible to estimate dangerous driving according to the driving score. As the degree of dangerous driving, for example, there are three levels: "accident", "near miss", and "sign of near miss". In that case, if it is a situation exceeding the first probability of dangerous driving occurrence, it is an "accident", if it is a situation exceeding the second probability of dangerous driving occurrence, it is a "near miss", and if it is a situation exceeding the third probability of dangerous driving occurrence, it is a "sign of near miss", and the degree of warning may be changed in each case. Note that a "near miss" is a case where sudden braking or sudden steering occurs due to a delay in visual recognition of an obstacle or a signal indication. A "sign of near miss" is a driving that does not involve sudden braking or sudden steering but may actually cause danger. This includes cases where a check for being involved has not been carried out.

[0043] From the above, by utilizing the driving support device described in the first embodiment, it is possible to appropriately output a warning to the driver and contribute to the safe driving of the driver and the improvement of driving skills. In order to improve the driver's skills, the time-series data of the actions taken by the driver later may be reviewed.

[0044] Moreover, this driving support device can not only be mounted on a vehicle, but also be mounted on a driving simulator to improve the driving skills of the driver.

[0045] Next, the driving support device 1 according to the second embodiment of the present invention will be described with reference to FIGS. 6 to 9. Regarding the common points with the first embodiment, duplicate explanations will be omitted.

[0046] Figure 6 is a functional block diagram of the driving support device 1 according to the present embodiment. As shown here, the driving support device 1 according to the present embodiment is different from the first embodiment in that the transition state generation unit 13 creates a transition state diagram representing the transition state as a diagram.

[0047] Figure 7 is a flowchart for explaining an operation example of the driving support device 1 during the running of the vehicle 21. Hereinafter, based on the driving environment of FIG. 2, each step of FIG. 7 will be sequentially explained. Since steps S71 to S74 are equivalent to steps S31 to S34 of FIG. 3, hereinafter, steps S75 and subsequent steps will be explained.

[0048] <<Step S75>> In step S75, the transition state generation unit 13 generates a transition state diagram. Here, the transition state diagram is a diagram that graphically represents the time-series driver behavior by taking the driving actions as nodes and the transitions between the actions as edges.

[0049] FIG. 8A is an example of a transition state diagram when an entrainment check is performed at an appropriate time in the "left turn at intersection scene" shown in FIG. 2. In this example, the driver who was looking straight ahead turns the steering wheel to the left after performing a rearview mirror check, a side mirror check, and an entrainment check. In this case, according to the state transition score table of FIG. 4, the scores are sequentially added as +1, +1, +1, +2, so the total driving score is 5.

[0050] On the other hand, FIG. 8B is an example of a transition state diagram when an entrainment check is not performed at an appropriate time in the "left turn at intersection scene" shown in FIG. 2. In this example, the driver who was looking straight ahead turns the steering wheel to the left after performing a rearview mirror check and a side mirror check without performing an entrainment check. In this case, according to the state transition score table of FIG. 4, the scores are sequentially added as +1, +1, +1, so the total driving score is 3. That is, since the driving score of FIG. 8B is lower than that of FIG. 8A, it can be evaluated that the driving skill of the driver in FIG. 8B is low.

[0051] <<Step S76>> In step S76, the driving score calculation unit 14 calculates a driving score based on the transition state diagram.

[0052] Here, we will explain the score calculation method using transition state diagrams, with reference to Figures 8A to 9.

[0053] Steps S76a to S76e perform the same operations as steps S35a to S35e in Figure 5 of Example 1.

[0054] In step S76f, the transition state diagram is dynamically modified based on the driver's past operating state. For example, as shown in Figure 8A, if a collision check is performed, the score attached to the edge is updated. In addition to updating the edge score, the graph structure may also be changed according to the time series by deleting nodes, deleting edges, etc.

[0055] <<Step S77>> In step S77, the calculation reason creation unit 15 stores the reason for calculating the score. At this time, exemplary driving behavior transitions may be stored using graph search theory or the like. Presenting exemplary driving behavior transitions to the driver contributes to improving driving skills.

[0056] <<Step S78>> In step S78, the calculation reason creation unit 15 outputs a warning to the driver based on the driving score.

[0057] Based on the above, in this embodiment, by using a transition state diagram, it is possible to easily handle whether the driver is appropriately performing safe actions, and to output warnings with high accuracy according to the probability of dangerous driving.

[0058] While embodiments of the vehicle control device according to this disclosure have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and any design changes, etc., that do not depart from the gist of this disclosure are also included in this disclosure.

[0059] 1 Driving support device, 11 Driver information acquisition unit, 12 Vehicle information acquisition unit, 13 Transition state generation unit, 14 Driving score calculation unit, 15 Calculation reason creation unit, 16 Output unit, 21 Vehicle, 22 Scene before entering intersection, 23 Scene inside intersection, 24 Scene leaving intersection, 25 Traffic light,

Claims

1. A driver assistance device that assists a driver in driving, comprising: a driver information acquisition unit that acquires driver information including the driver's gaze information; a vehicle information acquisition unit that acquires vehicle information of a vehicle driven by the driver; a transition state generation unit that generates transition states of the driver's actions using the driver information and the vehicle information; a driving score calculation unit that calculates a driving score for the driver's actions based on a correspondence table of the action transition states and scores generated by the transition state generation unit; a calculation reason creation unit that creates a reason for calculating the driving score calculated by the driving score calculation unit using the action transition states; and an output unit that outputs the driving score and the reason for calculating the driving score.

2. A driving assistance device according to claim 1, wherein the driver information acquisition unit acquires the driver's gaze information and head turn angle, and calculates the object to be viewed from the gaze information and head turn angle.

3. A driver assistance device according to claim 1, characterized in that the driving score is calculated based on the driver's past behavior.

4. The driving support device according to claim 1, wherein the output unit outputs exemplary driving in a time series.

5. A driving assistance device according to claim 1, characterized in that the output unit changes the warning level according to the magnitude of the driving score.

6. A driving support device according to claim 1, wherein the transition state generation unit generates a transition state diagram representing the transition states as diagrams, and the driving score calculation unit calculates a driving score for the driver's actions based on a correspondence table between the transition state diagram and the score.

7. The driver assistance device according to claim 6, characterized in that the states in the transition state diagram are defined as nodes and transitions as edges, and each edge has a driving score.

8. A driving assistance device according to claim 7, characterized in that the score of the edge is dynamically changed according to the transition state, and driving skill is determined by adding the dynamically changed driving scores in a time series.

9. A driving assistance method comprising: a driver information acquisition step of acquiring driver information including the driver's gaze information; a vehicle information acquisition step of acquiring vehicle information of a vehicle driven by the driver; a transition state generation step of generating transition states of the driver's actions using the driver information and the vehicle information; a driving score calculation step of calculating a driving score for the driver's actions based on a correspondence table of the action transition states and scores; a calculation reason creation step of creating a reason for calculating the driving score calculated using the action transition states; and an output step of outputting the driving score and the reason for calculating the driving score.

Images