Driving assistance device

Abstract

Driving assistance device (10) which is mounted in a vehicle (1) to assist a driver during a journey of the vehicle (1) by detecting a state of the driver, wherein the vehicle (1) is between a manual driving mode in which driving operations for acceleration, braking and steering are performed manually by the driver, and an automated driving state in which at least one of the driving operations is carried out automatically, is interchangeable, wherein the driving assistance device (10) comprises: a driver state monitoring unit (120) that detects whether the driver's state is an inattentive state; an alarm unit (140) which, when the driver's inattentive state is detected, alerts the driver; a driving unit (500) operated by the driver to carry out the driving operations; and a driving state change unit (130) which, when the driver operation of the driving operation unit (500) is detected during the automated driving state of the vehicle (1), changes at least one of the driving operations in the automated driving state to the manual driving state, where when at least one of the driving modes in the automated driving state is switched to the manual driving state by the driving state change unit (130), the driver state detection unit (120) detects whether the driver's state is an agitated state; and If the driver's agitated state is detected, the alerting unit (140) will alert the driver.

Description

TECHNICAL AREA

[0001] The present invention relates to a driver assistance technology used for a vehicle that can switch between an automated driving state and a manual driving state. The technology assists a driver in driving the vehicle by detecting the driver's state. STATE OF THE ART

[0002] Technologies have been developed and implemented that enhance vehicle safety by monitoring the driver's state based on factors such as the direction of the driver's line of sight and biological information (e.g., eyelid behavior and heart rate). These technologies alert the driver if inattentiveness is detected. Furthermore, in recent years, vehicles have been introduced that automatically perform some driving operations (e.g., driving, turning, and stopping) to reduce driver workload. It is also anticipated that fully autonomous vehicles capable of accelerating, steering, and braking, while allowing human drivers to take over in emergencies, will soon see practical application.

[0003] One objective of automated driving is to reduce driver workload. Therefore, it is conceivable that if drivers place greater trust in automated driving, they will delegate more driving operations to it, thereby accepting greater risks. As a result, there is a concern that drivers will become too relaxed and inattentive while driving. If a driver becomes inattentive and is no longer able to take action in an emergency, safety cannot be guaranteed. Thus, the proposed technologies for reducing driver workload and ensuring driving safety also monitor the driver's state during automated driving. If it is detected that the driver is inattentive to an intolerable degree, the driver is alerted (JP 2006-318446A and JP 2013-41524A).The tolerable level of driver inattention during automated driving should not differ significantly from that during manual driving, in order to ensure safe driving. However, this could diminish the benefits of automated driving. Therefore, when proposing the technologies in JP 2006-318446A and JP 2013-41524A, various studies were conducted regarding the maximum tolerable level of driver inattention during automated driving.

[0004] DE 101 03 401 A1 discloses a driving assistance device that is mounted in a vehicle to assist a driver while driving the vehicle by detecting the driver's state, wherein the vehicle is switchable between a manual driving state in which driving operations for acceleration, braking and steering are performed manually by the driver, and an automated driving state in which at least one of the driving operations is performed automatically, wherein the driving assistance device comprises: a driver state detection unit that detects whether the driver's state is an inattentive state; an alarm unit that, when the inattentive state of the driver is detected, alerts the driver;a driving unit operated by the driver to carry out driving operations, wherein, if during automated driving mode the system determines that a switch to manual driving mode is required due to a hazardous situation, the driver is prompted accordingly, and if the driver does not respond within a time period dependent on the driver workload, a further warning is issued.

[0005] US 2012 / 0 083 960 A1 describes a system for predicting the behavior of externally detected objects in autonomously driving vehicles.

[0006] DE 103 43 178 A1 describes a driver assistance system with a variable longitudinal guidance strategy, comprising a sensor device for detecting the traffic environment and a controller for regulating the acceleration of the vehicle according to a predetermined, variably determinable longitudinal guidance strategy.

[0007] DE 695 30 459 T2 describes a detection device for detecting a deterioration of a mental state while driving, such as drowsiness, fatigue and impatience, which occur in a driver, based on physiological data of the driver and road driving data derived from a navigation system, and for generating a corresponding alarm. SUMMARY OF THE INVENTION

[0008] However, according to studies conducted by the inventor of this application, regardless of the maximum tolerable level of driver inattention set, ensuring driving safety is difficult for the following reasons. Since it is true that a driver can more easily become inattentive during automated driving than during manual driving, it is important to set an appropriate maximum tolerable level of inattention to ensure driving safety. However, during automated driving, there is a possibility that the driver may enter a state of inattention where ensuring safe driving is difficult, a state that presumably does not occur during manual driving. None of the technologies proposed to date take this possibility into account.If the driver enters such a dangerous situation, which is unique to automated driving, it is difficult to ensure a safe journey.

[0009] The present invention arose in light of the above, and it is an object of the invention to provide a driving assistance device and a driving assistance method for ensuring the safe operation of a vehicle by detecting the driver's state during automated driving. This object is achieved by a driving assistance device with the features of claim 1 and claim 6, respectively, and by a driving assistance method with the features of claim 5. The dependent claims are directed to advantageous embodiments of the invention.

[0010] A driving assistance device according to an example of the present invention is mounted in a vehicle to assist a driver during a journey of the vehicle by detecting a state of the driver, wherein the vehicle is switchable between a manual driving state in which driving operations for acceleration, braking and steering are carried out manually by the driver and an automated driving state in which at least one of the driving operations is carried out automatically.The driving assistance device comprises: a driving state detection unit that detects whether the driver is inattentive; an alerting unit that alerts the driver when inattentiveness is detected; a driving operation unit operated by the driver to perform driving operations; and a driving state switching unit that, when driver operation of the driving operation unit is detected during the vehicle's automated driving state, switches at least one of the driving operations in the automated driving state to the manual driving state. When at least one of the driving operations in the automated driving state is switched to the manual driving state by the driving state switching unit, the driver state detection unit detects whether the driver is agitated. If agitatedness is detected, the alerting unit alerts the driver.

[0011] A driving assistance method according to an example of the present invention is used for a vehicle to assist a driver during a journey of the vehicle by detecting a state of the driver, wherein the vehicle is switchable between a manual driving state in which driving operations for acceleration, braking and steering are carried out manually by the driver and an automated driving state in which at least one of the driving operations is carried out automatically.The driver assistance procedure includes: a driver state detection process that detects whether the driver's state is inattentive; an alerting process that alerts the driver when the driver's inattentive state is detected; and a driving state change process which, when the driver operation of the driving unit is detected during the vehicle's automated driving state, changes at least one of the driving operations in the automated driving state to the manual driving state, wherein, when at least one of the driving operations in the automated driving state is changed to the manual driving state in the driving state change process, the driver state detection process detects whether the driver's state is agitated; and the alerting process alerts the driver when the driver's agitated state is detected.

[0012] It is generally assumed that a driver tends to enter a state of inattentiveness during automated driving. In this respect, compared to manual driving, automated driving tends to be conducted in compliance with traffic regulations in every situation. Therefore, automated driving that continues for a long time can cause stress for the driver. There are instances where, as a result of increasing stress, the driver, wishing to relieve it, attempts to switch to manual driving mode by operating at least one of the driving modes in the automated driving state. Since the driver may be in a state of agitation in a situation like this, the system detects whether the driver is in an agitated state.If it is detected that the driver is agitated, the driver is alerted. In this way, even if the driver enters a dangerous state specific to automated driving, the driver can be calmed down, thus ensuring a safe journey. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and further problems, features and advantages of the present invention will become clear with reference to the following detailed description and the accompanying drawings. These show: Fig. 1A a diagram representing a vehicle in which a driving assistance device according to an embodiment of the present invention is mounted; Fig. 1B a diagram illustrating a configuration of the driving assistance device according to the embodiment; Fig. 2 a diagram that presents basic measurements for assessing a driver's condition; Fig. 3 a flowchart of a driver assistance processing system according to the embodiment; Fig. 4A a diagram that conceptually illustrates an example of how the driver's psychology changes while the vehicle is in motion; Fig. 4B a diagram that conceptually represents another example of how the driver's psychological state changes while the vehicle is in motion; and Fig. 5 a flowchart of a driver assistance processing system according to a modification of the embodiment. FORM OF EXECUTION FOR IMPLEMENTING THE INVENTION

[0014] The following describes an embodiment of the present invention. A. Device configuration

[0015] Fig. Figure 1A shows a vehicle 1 in which a driving assistance device 10 of the present embodiment is mounted. As shown in Fig. As shown in Figure 1A, the vehicle 1 is equipped with a control device 100, which forms a main part of the driving support device 10 of the present embodiment, an automated driving device (device for automated driving) 400 for automatically driving the vehicle 1 and a vehicle navigation system.

[0016] During manual driving, a driver operates the vehicle 1, for example, by steering a steering wheel 500 and operating an accelerator and a brake pedal (not shown). After the driving state has switched to automated driving, the automated driving device 400 operates the steering wheel 500, the accelerator pedal, and the brake pedal for the driver. The steering wheel 500 has a shaft attached to an actuator 502, which has a built-in steering angle sensor. The automated driving device 400 operates the steering wheel 500 by driving the actuator 502 such that the steering angle sensor output equals a setpoint. Actuators (not shown) are also attached to the accelerator and brake pedals, which have built-in sensors for detecting the magnitudes of the accelerator and brake pedal movements.The 400 automation drive device detects accelerator pedal and brake operating parameters based on the outputs of the respective sensors in order to operate the accelerator pedal and brake, and drives the respective actuators.

[0017] The automated driving device 400 of the present embodiment is described as a device that automatically performs the accelerator pedal operation for accelerating / decelerating the vehicle 1, the brake operation for braking the vehicle 1, and the steering wheel operation for steering the vehicle 1 for the driver. Alternatively, the automated driving device 400 can be a device that automatically performs some of the above operations.

[0018] A driver camera 200 for photographing the driver's face is mounted on the vehicle 1. Images captured by the driver camera 200 are input into the control device 100. Light in a near-infrared wavelength range is primarily used when photographing the driver's face. In the present embodiment, the driver also wears an electrocardiography sensor 210 (referred to as "Sensor A"). Fig. 1B) and a blood pressure sensor 220 (referred to as "Sensor B" in Fig. (designated 1B). Data on the driver's heart rate and blood pressure are wirelessly entered into the control unit 100.

[0019] The control device 100 detects the driver's condition based on driver facial image data and driver heart rate and blood pressure data. For example, if it is determined that the driver is in an inattentive state, the control device 100 alerts the driver, for example, by emitting various speech or sound effects from a loudspeaker 300 or by activating a seat vibration device 320 built into the driver's seat. The control device 100 of the present embodiment can exchange data with the automated driving device 400 and the vehicle navigation system 402. Therefore, even if the driver enters a dangerous condition, which can occur particularly during automated driving, the control device 100 can alert the driver to their current condition and, as a result, ensure driving safety.

[0020] Fig. 1B outlines the internal configuration of the control device 100. As shown in Fig. As shown in Figure 1B, the control device 100 of the present embodiment includes a sensor communication unit 110, a driver state detection unit 120, a driving state change unit 130, an alarm unit 140, and a driving area detection unit 150. Note that these five units each represent abstract concepts obtained by appropriately classifying the interior of the control device 100, focusing on the control device 100's function of detecting the driver state and alarming the driver. That is, the five units do not represent physically separate sections of the control device 100. Each of the units can be implemented either by software, through a computer program to be executed by the control device 100, which contains a processor and an I / O unit, or by hardware, using an integrated circuit such as an LSI.Of course, it is possible to implement the units by combining software and hardware. The computer program is stored in a non-volatile storage medium such as semiconductor memory.

[0021] The sensor communication unit 110 communicates wirelessly (or via cable) with the electrocardiography sensor 210 and the blood pressure sensor 220 and collects driver data, for example data about a driver's electrocardiogram and pulse rate, and outputs such data to the driver condition monitoring unit 120.

[0022] Image data relating to the driver's face, photographed by the driver camera 200 (facial image data), are entered into the driver state monitoring unit 120 without the involvement of the sensor communication unit 110 and analyzed in a facial image analysis unit 122, which is contained within the driver state monitoring unit 120. Based on the received facial image, the facial image analysis unit 122 determines the driver's facial direction, blink rate, and eye opening degree.

[0023] Data on the driver's facial orientation, turn signal frequency, and eye opening, as well as heart rate and blood pressure data obtained from the driver's electrocardiogram and pulse rate, provide measurements used as baseline data for assessing the driver's condition (e.g., inattentiveness). These measurements are referred to as "baseline measurements" below. The baseline measurements are described in more detail with reference to the diagrams.

[0024] The driver state monitoring unit 120 monitors the driver's state, for example, an inattentive state, based on baseline measurements. To monitor the driver's state, a comparison is made between the baseline measurements and reference values ​​for driver state monitoring (hereinafter referred to as "monitoring reference values"). The monitoring reference values ​​are pre-stored in a memory unit 124, which is contained within the driver state monitoring unit 120.

[0025] If it is detected that the driver is in an inattentive state, the driver state detection unit 120 communicates this detection result to an alarm unit 140. The alarm unit then alerts the driver by emitting a pre-recorded speech or sound effect through the loudspeaker 300 or by vibrating the driver's seat using the seat vibration device 320. However, the driver alerting procedure is not limited to the emission of speech or sounds or vibration of the driver's seat. For example, the driver can be alerted by cold air blown onto the driver using an air conditioning system (not shown).

[0026] The control device 100 also includes the driving state change unit 130. The driving state change unit 130 exchanges data with the automated driving device 400 and can detect an automated driving state of the vehicle 1, information about control setpoints for the automated driving of the vehicle 1 by the automated driving device 400, and similar information. Furthermore, the driving state change unit 130 is coupled to the steering angle sensor, which is integrated into the actuator 502 of the steering wheel 500.

[0027] If, in an automated driving mode, the output of the steering angle sensor deviates significantly from a corresponding control setpoint configured for the automated driving device 400, the control device 100 determines that the steering wheel 500 is operated during automated driving. The control device 100 then sends a signal to the automated driving device 400 to switch the operation of the steering wheel 500 (steering operation) from an automated driving mode to a manual driving mode.If, during automated operation, the magnitude of the operation of the accelerator pedal or the brake pedal deviates significantly from a corresponding control setpoint set for the automation operating device 400, the control device 100 outputs a signal to the automation operating device 400 to switch the operation of the accelerator pedal or brake pedal (accelerator pedal or brake operation) from an automated driving state to a manual driving state.

[0028] During automated driving, the driving operation is temporarily (or for a predetermined duration) switched from an automated driving state to a manual driving state as a result of the driver's operation of the steering wheel, accelerator pedal, or brake pedal, without performing a normal switching procedure. This event is generally referred to as "overriding".

[0029] The control device 100 of the present embodiment also includes the driving area detection unit 150. The driving area detection unit 150 is coupled to the vehicle navigation system 402. Therefore, information about whether automated driving is permitted in the road segment where the vehicle 1 is currently traveling can be obtained from the vehicle navigation system 402. Although there are many urban road segments where automated driving is not permitted, automated driving is permitted for motor vehicles only on most sections of highways. The vehicle navigation system 402 stores information regarding whether automated driving is permitted in advance for all segments of most roads. The driving area detection unit 150 receives such information through communication with the vehicle navigation system 402.

[0030] When a route has been set in the vehicle navigation system 402, information can be obtained not only regarding whether the current road section is one where automated driving is permitted, but also other information, such as whether there is any road section ahead where automated driving is not permitted (Automated Driving Prohibition Section), and if one exists, what the distance to the Automated Driving Prohibition Section is.

[0031] Although the present embodiment is presented under the assumption that information on whether automated driving is permitted is stored in the vehicle navigation system 402 for individual road segments, an alternative method can be used. For example, the vehicle 1 can obtain such information from an external database using a wireless communication device (not shown) or by recognizing road signs. Alternatively, for individual road segments that include loop road sections at complex junctions, where it is difficult to determine solely using the vehicle navigation system, the vehicle can determine whether automated driving is permitted based on vehicle behavior.

[0032] In the present embodiment, the driver state detection unit 120 corresponds to a driver state detection device. The driving state change unit 130 corresponds to a driving state change device. The alarm unit 140 corresponds to an alarm device. The driving area detection unit 150 corresponds to a driving area detection device. The steering wheel 500 corresponds to a driving control unit.

[0033] Fig. Section 2 outlines data (basic measurements) used by the driver state monitoring device 120 of the present embodiment. As described above, the basic measurements include, for example, measurements of face direction, blink rate, eye opening degree, heart rate, and blood pressure.

[0034] For the face direction, a duration is recorded during which the face direction, detected from a facial image, and the direction of travel of the vehicle 1, detected by a steering angle sensor (not shown), do not match. When a driver is inattentive, they tend not to look sufficiently in the direction of travel or to tilt their head. Therefore, if the duration during which the driver's face direction and the vehicle's direction of travel do not match is recorded, and if this duration exceeds a predetermined time, it can be determined that the driver is in an inattentive state.

[0035] To determine facial direction from a facial image, a known method can be used. For example, feature points of facial parts, such as the inner and outer corners of the eyes, nose, and mouth, are extracted from the facial image, and the direction of the face is determined based on the positional relationship between the extracted feature points.

[0036] The blink rate is determined by analyzing driver face images photographed at a predetermined period (e.g., 30 ms) and, based on the detected eyelid behavior, the number of blinks per minute is recorded.

[0037] The eye-opening rate is measured as the proportion of time per minute that the driver's eyes are open, based on eyelid behavior (the behavior of the upper eyelids). If a driver is inattentive, their blink rate and eye-opening rate decrease. When these values ​​reach their respective lower limits, it can be determined that the driver is in an inattentive state.

[0038] For heart rate, the number of heartbeats per minute of the driver is recorded, and for blood pressure, maximum and minimum blood pressures are recorded. If a driver is inattentive, their heart rate and blood pressure decrease. Therefore, if the driver's heart rate and blood pressure reach their respective lower limits, it can be determined that the driver is in an inattentive state. Another known method can be used to detect a driver's inattentive state.

[0039] The method of assessing driver condition was described, focusing on the driver's inattentive state. Similarly, if the driver is tired, their condition can be assessed, for example, by focusing on their poor physical condition.

[0040] The objects used for basic measurements are not limited to those mentioned above. For example, the driver's pulse rate, brain waves, respiration, body movement, and body temperature can be recorded as basic measurements. In this case, the Driver State Monitoring Unit 120 records such basic measurements using, for example, a pulse sensor, a brain wave sensor, a respiration sensor, a body movement sensor, and a thermometer. B. Driving assistance processing

[0041] Fig. Figure 3 is a flowchart of a driver assistance processing operation performed by the control device 100 of the present embodiment. The driver assistance processing is initiated when the combustion engine of the vehicle 1 is started and is performed regardless of whether the vehicle 1 is in a manual or automated driving state. As will be described in more detail later, suitable detection reference values ​​for detecting the driver's state are set depending on whether the vehicle 1 is in a manual or automated driving state, thus ensuring safe driving even if the driver enters a hazardous state specific to automated driving.

[0042] In the driver assistance processing, at the beginning, as is described in Fig. As shown in Figure 3, driver facial image data captured by the driver camera 200 is received, and signals relating to the driver's heart rate and blood pressure are also received from the electrocardiography sensor 210 and the blood pressure sensor 220 (S100). Subsequently, the driver state monitoring unit 120 acquires the baseline measurements by analyzing the received data (S102).

[0043] The baseline measurements acquired after the combustion engine is started (or within a predetermined time period after the combustion engine is started) are used to calculate acquisition reference values ​​for detecting the driver's state. In the present embodiment, three types of acquisition reference values ​​are set. There are acquisition reference values ​​A for detecting an inattentive state of the driver during manual driving, acquisition reference values ​​B for detecting an inattentive state of the driver during automated driving, and acquisition reference values ​​C for detecting an agitated state of the driver during automated driving. Considering that the three types of acquisition reference values ​​may vary between individuals, acquisition reference values ​​for the driver of vehicle 1 are calculated based on the baseline measurements acquired in S102 (S104).

[0044] If the acquisition reference values ​​are calculated using the baseline measurements acquired within a predetermined time period after the start of the internal combustion engine, it is possible to calculate the acquisition reference values ​​appropriately even if acquiring the baseline measurements becomes temporarily impossible for any reason.

[0045] The following section first outlines the three types of reference values ​​for data collection, and then describes methods for calculating the three types of reference values ​​for data collection, taking into account differences between individuals.

[0046] The detection reference values ​​A for detecting a driver's inattentive state during manual driving are as follows. It is generally known that when a driver becomes inattentive, their turn signal frequency decreases. The driver's eye opening, heart rate, and blood pressure also decrease when the driver becomes inattentive. Regarding gaze direction, when the driver becomes inattentive, it becomes difficult for them to look ahead in the direction of travel, and as a result, the time during which the driver's gaze direction differs from the vehicle's direction of travel (off-look time) increases. Alternatively, when the driver becomes inattentive, their behavior regarding the perception of the surrounding situation for safety purposes decreases, and there is a tendency for the driver's gaze direction to remain unchanged.

[0047] Therefore, a driver entering an inattentive state can be detected by recording a decrease in the measured values ​​of turn signal frequency, eye opening degree, heart rate, and blood pressure. With regard to facial direction, a driver entering an inattentive state can be detected by recording an increase in the duration for which the driver is looking away or whose facial direction remains unchanged.

[0048] As described above, the detection reference values ​​A are used to detect an inattentive state of the driver during manual driving.

[0049] During manual driving, even a slight lapse in driver attention can cause the vehicle to enter a dangerous state. However, during automated driving, a driver's lapse into a slightly inattentive state should be tolerated, as the automated driving device 400 is driving the vehicle for the driver. If, however, the driver becomes extremely inattentive to the point where, for example, they cannot immediately take hazard avoidance action in an emergency, they will not be able to avert danger by switching from automated to manual driving. Therefore, it is conceivable that the driver's inattentive state detected during automated driving should be considered more serious than the driver's inattentive state detected during manual driving.

[0050] As described above, the detection reference values ​​B are used to detect a driver inattentive state during automated driving. Because the driver inattentive state to be detected during automated driving is more serious than the driver inattentive state to be detected during manual driving, the detection reference values ​​B set for blink rate, eye opening level, heart rate, and blood pressure are lower than the corresponding detection reference values ​​A. With respect to the driver's face direction, the detection reference value B to be set is higher than the detection reference value A.

[0051] Furthermore, when driving automatically, compared to driving manually, it's necessary to consider the possibility that the driver might become agitated. One reason for this is as follows: Automated driving is generally not only conducted to comply with traffic regulations but also to maintain safety. If the driver is in a hurry or irritated for any reason, they may feel stressed during automated driving. For example, when following a vehicle ahead during automated driving, a good distance is maintained. This often allows other vehicles to merge in front of the automated vehicle. In such a situation, it's conceivable that a driver in a hurry might feel stressed and irritated, consequently entering a state of agitation.

[0052] When the driver is agitated, it is assumed that the blink rate, eye opening rate, heart rate, and blood pressure increase compared to a case where the driver is in an inattentive state. Regarding gaze direction, the driver is likely to lose the composure to check the surroundings and tend to look away from the direction of travel of the vehicle. As a result, the time the driver's gaze direction differs from the vehicle's direction of travel (off-gaze time) and the time the driver's gaze direction remains unchanged are assumed to be extremely reduced.It is therefore conceivable that the driver's entry into an agitated state can be detected by recording the increase in the blink rate, eye opening degree, heart rate and blood pressure of the driver and a decrease in the off-look time or a stationary time recorded on the basis of the direction of the face.

[0053] As described above, the detection reference values ​​C are used to detect an agitated state of the driver during automated driving.

[0054] The three types of measurement reference values ​​A to C are expected to differ between different individuals. In S104, which is in Fig. As shown in Figure 3, the three types of recording reference values ​​A to C are determined as follows.

[0055] First, the acquisition reference values ​​A are calculated based on the baseline measurements recorded in S102. That is, the acquisition reference values ​​A for blink rate, eye opening degree, heart rate, and blood pressure are calculated by subtracting corresponding predetermined values ​​from the baseline measurements recorded in S102. In this way, suitable acquisition reference values ​​A can be determined while taking into account differences between individuals. Regarding face direction, it is conceivable that no individual difference needs to be considered, so that a predetermined value (for example, 3 seconds) is determined as an acquisition reference value A.

[0056] The reference values ​​B and C are calculated based on the reference values ​​A. That is, the reference values ​​B for blink rate, eye opening degree, heart rate, and blood pressure are calculated by subtracting the corresponding predetermined values ​​from the corresponding reference values ​​A. The reference values ​​C for blink rate, eye opening degree, heart rate, and blood pressure are calculated by adding the corresponding predetermined values ​​to the corresponding reference values ​​A.

[0057] The reference value B for facial direction is calculated by adding the reference value A for facial direction to a corresponding predetermined value. The reference value C for facial direction is calculated by subtracting a corresponding predetermined value from the reference value A for facial direction.

[0058] In S104, which is in Fig. As shown in Figure 3, the three types of acquisition reference values ​​A to C, which were calculated as described above, are stored in the memory unit 124, which is contained in the driver state acquisition unit 120.

[0059] Subsequently, in the driver assistance processing, which is in Fig. As shown in section 3, the driver's face image data and signals relating to the driver's heart rate and blood pressure are received again (S106), and then the baseline measurements taken in Fig. 2 are listed, recorded (S108).

[0060] The system then determines whether vehicle 1 is in an automated driving state (S110). The driving state change unit 130, which is located in Fig. As shown in 1B, it exchanges data with the automated driving device 400, so that the control device 100 can determine whether the vehicle 1 is in an automated driving state.

[0061] If vehicle 1 is not in an automated driving state (S110: No), it is assumed that vehicle 1 is in a manual driving state. Therefore, the driver state detection unit 120 selects detection reference values ​​A (S112) from the three types of detection reference values ​​calculated in S104. As described above, the three types of detection reference values ​​calculated in S104 are stored in the memory unit 124.

[0062] If the vehicle 1 is in an automated driving state (S110: Yes), the driver state detection unit 120 selects the detection reference values ​​B from the three types of detection reference values ​​stored in the memory unit 124 (S114).

[0063] The control device 100 then determines whether an override is detected. As described above, an "override" refers to a change in driving mode from an automated driving state to a manual driving state as a result of driver operation during automated driving, for example, steering wheel operation 500. As described above with reference to Fig. As described in section 1, the driving state change unit 130 can detect whether the driver has caused an override by comparing the output of the steering angle sensor contained in the actuator 502 with a corresponding control setpoint set for the automated driving device 400.

[0064] Overriding occurs only during automated driving. Therefore, after the acquisition reference values ​​A have been selected for use in a manual driving state (S112), no determination is made as to whether overriding has occurred. Only after the acquisition reference values ​​B have been selected (S114) is it determined whether overriding has occurred (S116).

[0065] If an override is detected (S116: Yes), an automation drive change signal is sent from the drive state change unit 130 to the automation drive device 400 (S118) to temporarily (or for a predetermined duration) switch the driving mode in an automated driving state. As a result, the driving mode in an automated driving state is switched to a manual driving state. When the driving mode is switched to a manual driving state, either all driving modes in an automated driving state can be switched to a manual driving state, or only the driving mode that has been detected as being performed manually by the driver can be switched to a manual driving state.

[0066] If, as described above, an override is detected (S116: Yes) and a transition to an automated driving state is initiated (S118), the driver state detection unit 120 selects the detection reference values ​​C (S120) from the three types of detection reference values ​​stored in the memory unit 124. That is, when an override is detected, the detection reference value is changed from the detection reference values ​​B selected in S114 to the detection reference values ​​C. As described above, the detection reference values ​​C are used to detect an agitated state of the driver during automated driving.

[0067] If no override is detected (S116: No), neither the processing to output an automation drive change signal to the automation drive device 400 (S118) nor the processing to select the detection reference values ​​C (S120) is carried out.

[0068] If the acquisition reference values ​​A, B or C calculated in S104 are selected (S112, S114, S120), it is determined whether the baseline measurements previously acquired in S108 reach the selected acquisition reference values.

[0069] In a case where acquisition reference values ​​A or B have been selected, the baseline reference values ​​for blink rate, eye opening degree, heart rate, and blood pressure are considered to be greater than the values ​​set as the corresponding acquisition reference values. In such cases, S122 therefore determines whether any of the baseline measurements have decreased to the corresponding acquisition reference value. With regard to facial direction, the baseline measurement is considered to be less than the corresponding set acquisition reference value, and thus S122 determines whether the baseline measurement has increased to the corresponding acquisition reference value.

[0070] In cases where the acquisition reference values ​​C have been selected, the baseline reference values ​​for blink rate, eye opening degree, heart rate, and blood pressure are considered lower than the values ​​set as the corresponding acquisition reference values. Therefore, in such cases, S122 determines whether any of the baseline measurements have increased to the corresponding acquisition reference value. With regard to facial direction, the baseline measurement is considered higher than the corresponding set acquisition reference value C, and thus S122 determines whether the baseline measurement has decreased to the corresponding acquisition reference value.

[0071] In the present embodiment, if any of the baseline measurements of blink rate, eye opening degree, heart rate, blood pressure, and face direction reaches the corresponding detection reference value, it is determined that the driver is in an inattentive or agitated state. Alternatively, it can be determined that the driver is in an inattentive or agitated state based on the results of the determination for several baseline measurements.

[0072] For example, it can be determined that the driver is in an inattentive or agitated state if a predetermined number of baseline measurements reach the corresponding detection reference values. Alternatively, it can be determined that the driver is in an inattentive or agitated state if, within a specific group of baseline measurements (e.g., heart rate and blood pressure), each baseline measurement reaches the corresponding detection reference value. In this way, it is possible to determine with greater reliability whether the driver is in an inattentive or agitated state.

[0073] If, as a result of comparing the baseline measurements with the acquisition reference values, as described above, it is determined that no baseline measurement has reached a corresponding acquisition reference value (S122: No), the processing returns to S106 to receive new image data of the driver's face and signals relating to the driver's heart rate and blood pressure. Based on the newly received data and signals, new baseline measurements are acquired (S108), and the subsequent processing steps are carried out as described above.

[0074] If it is determined that any baseline measurement has reached a corresponding recording reference value (S122: Yes), the driver will be alerted according to the corresponding recording reference value (S124).

[0075] For example, in cases where the detection reference values ​​A have been selected, an alarm tone is emitted from the loudspeaker 300, or cold air from an air conditioning unit (not shown) is blown onto the driver, thus causing the driver to emerge from a state of inattentiveness. Alternatively, the driver's seat can be vibrated using a seat vibration device 320.

[0076] In cases where the detection reference values ​​B were selected, because it is assumed that vehicle 1 is in an automated driving state, the alarm tone is emitted and cold air is blown out in a more moderate manner than in cases where the detection reference values ​​A were selected.

[0077] In cases where the detection reference values ​​C have been selected, it is assumed that the driver is in an agitated state. In such cases, a tone is emitted from loudspeaker 300 to help the driver relax, or a gently alarming voice is broadcast. Additionally, the air conditioning system blows out cooler air than in other cases to cool the driver.

[0078] By alerting the driver in the above manner, a driver who is in an inattentive state can be brought into an attentive state, and a driver who is in an agitated state can be calmed down.

[0079] After the driver is alerted (S124), processing returns to S106 and repeats the subsequent processing steps as described above.

[0080] In the present embodiment, S122 and S224, which are performed by the control device 100, correspond to a driver state detection process. S124 and S226, which are performed by the control device 100, correspond to an alarm process. S118 and S218, which are performed by the control device 100, correspond to a driving state change process.

[0081] By implementing the driving assistance processing of the present embodiment described above, it is possible to control the driver's psychological state and ensure the driving safety of vehicle 1. This will be described in more detail below.

[0082] Fig. Section 4A conceptually illustrates how the driver's psychology changes while the vehicle 1 is in motion. During a manual driving segment, the driver must operate the vehicle 1 manually by manipulating the steering wheel, brake pedal, and accelerator pedal while monitoring the surrounding environment. During extended periods of driving under such conditions, the driver's alertness decreases, and the driver becomes inattentive. As the driver's inattentiveness increases, driving safety is compromised. Therefore, when the driver's inattentiveness reaches a certain level (low level), the driver is alerted to the situation at time t1. Fig. 4A is shown. As a result, the driver's alertness returns to a normal state.

[0083] When vehicle 1 enters an automated driving section (a section of road where automated driving is permitted) and an automated drive is initiated, the automated driving device 400 drives vehicle 1 for the driver, thus reducing the driver's workload. As a result, it is easier for the driver to become inattentive. Since automated driving is designed to reduce the driver's workload, a degree of driver inattentiveness is considered tolerable. However, if the driver becomes extremely inattentive, for example, to such an extent that the driver cannot immediately take hazard avoidance action even in an emergency, it becomes difficult to ensure safety in an emergency, even when attempting to switch from automated to manual driving.Thus, during automated driving, the driver is not alerted compared to manual driving if the driver's inattention is of a minor degree, but is alerted if the driver's inattention increases significantly (up to a serious degree), as at time t2, which is in . Fig. When 4A is shown, the driver is alerted.

[0084] In this way, during automated driving, the driver's mind is kept in an inattentive state such that, while relaxed to a certain degree, the driver is able to switch from automated driving to manual driving in an emergency to ensure safety (see Fig. 4A).

[0085] Compared to manual driving, traffic rules are followed more strictly during automated driving. Therefore, if automated driving continues for an extended period, the driver may feel stressed. For example, during automated driving, a safe distance is maintained from the vehicle ahead. This allows other vehicles to easily cut in front of the automatically driven vehicle, causing the driver of the automated vehicle to feel stressed. Due to this stress, the driver, who was initially relaxed, may feel tense. This tendency is more noticeable if the driver is in a hurry or irritated for any reason.

[0086] Fig. Section 4B conceptually illustrates how the duration of automated driving over an extended period causes stress for the driver, and how this stress shifts the driver's psyche from a relaxed to a tense state. It is conceivable that if the driver's tension rises above a certain level, the driver will override the automated system to relieve the stress. In this case, the driver is in an agitated state, in contrast to the inattentive state recorded during automated driving. In such a situation, the driver is very likely to, for example, operate the steering wheel and accelerator pedal roughly. Or the driver is likely to drive very recklessly.

[0087] Therefore, if the driver overrides the automated driving function, it is determined at time t3 whether the driver is agitated, as described in Fig. 4B is shown. If it is determined that the driver is agitated, a voice message, for example “Aren’t you a little irritated?”, is played to alert the driver. In this way, it is possible to calm the driver, as shown in Fig. 4B is shown to prevent the driver from driving in a dangerous manner.

[0088] After an override is detected in S116 (S116: Yes), the driver's agitation status is determined and reassessed until a predetermined time has elapsed. If the driver, once alerted, has not calmed down, this allows for a second alert. Furthermore, in cases where the driver's level of agitation immediately following an override detection does not warrant an alert, but subsequently becomes agitated to the point where an alert is necessary, the driver can be alerted to calm down.Since, after an override is detected, a determination of whether the driver is agitated is repeated until a predetermined time period has elapsed, even in cases where the driver's agitated state cannot be correctly detected immediately after an override for any reason, the driver's agitated state can eventually be detected and the driver can be alerted.

[0089] In the embodiment described above, an agitated state of the driver is detected after the driver has overridden the system. Since it is conceivable that the driver overrides after becoming stressed and entering an agitated state, it can be assumed that the detection of the driver's agitated state can be performed before the override occurs. However, in the present embodiment, the detection of the driver's agitated state is performed after the driver has overridden the system. The reason for this is as follows.

[0090] Initially, the occurrence of an override means that vehicle 1 is driven automatically until the override occurs. Even if the driver is under considerable stress, as long as vehicle 1 is driven automatically, driving safety is not immediately threatened. In such a situation, driving safety is only potentially threatened.

[0091] However, if an override leads to a switch from automated to manual driving, the potential threat to driving safety increases significantly. Taking this into account, the present embodiment determines, after detecting an override, whether the driver is agitated. If it is determined that the driver is agitated, safety measures are taken (the driver is alerted).

[0092] It goes without saying that detecting a driver's state of agitation and alerting the driver before they override the voice control should not be prevented. For example, if a highly agitated state of the driver is detected, the driver can even be alerted before they override the voice control as a precautionary measure. C. Modification

[0093] In the embodiment described above, whenever an override occurs during automated driving, it is determined whether the driver is agitated. Alternatively, if an override occurs during automated driving, it is detected in a predetermined situation whether the driver is in an agitated state. This modification is briefly described, focusing on one difference between the embodiment above and the modification. When describing the modification, configurations of the modification that are similar to those of the embodiment above are designated with the same reference numerals as in the embodiment above, and a detailed description of such similar configurations is omitted.

[0094] Fig. Figure 5 is a flowchart of a driver assistance processing operation of the present modification. When the driver assistance processing is started, the control device 100 of the present modification receives driver face image data, similar to the embodiment described above, which is captured by the driver camera 200, and receives signals relating to the driver's heart rate and blood pressure from the electrocardiography sensor 210 and the blood pressure sensor 220 (S200). Subsequently, baseline measurements are acquired by analyzing the received data (S202).

[0095] The three types of acquisition reference values ​​A to C mentioned above are then calculated (S204). The contents of the acquisition reference values ​​A to C and their calculation method are the same as described in connection with the embodiment above, and therefore their description is not repeated. The calculated acquisition reference values ​​A to C are stored in memory unit 124.

[0096] Subsequently, the driver's facial image data and signals relating to the driver's heart rate and blood pressure are received again (S206). Based on the received data and signals, new baseline measurements are recorded (S208).

[0097] The system then determines (S210) whether vehicle 1 is in an automated driving state. If vehicle 1 is not in an automated driving state (S210: No), the acquisition reference values ​​A for a manual driving state are selected (S212).

[0098] If vehicle 1 is in an automated driving state (S210: Yes), the acquisition reference values ​​B for an automated driving state are selected (S214).

[0099] The control device 100 then determines whether an override is detected (S216). If it is determined that an override is detected (S216: Yes), an automated driving change signal to switch from automated driving to manual driving is sent from the driving state change unit 130 to the automated driving device 400 (S218). As a result, the driving operation is switched from an automated driving state to a manual driving state.

[0100] In the driver assistance processing of the present modification, after the automated driving change signal (S218) has been sent, it is determined (S220) whether the current road segment where automated driving is permitted (automated driving permission segment) will soon end. As described in Fig.As shown in Figure 1B, the driving area detection unit 150, which is contained in the control device 100, can receive information from the vehicle navigation system 402 regarding whether automated driving is permitted in the road segment where the vehicle 1 is currently driving, whether there is any road segment ahead where automated driving is not permitted (automated driving prohibition section), and if so, the distance to the automated driving prohibition section. Based on this information, S220 determines whether the current road segment where automated driving is permitted will soon end (for example, end at a location approximately 2 km ahead).

[0101] If the result is determined that the current road section where automated driving is permitted will not end soon (will continue) (S220: No), the detection reference values ​​C are selected to detect an agitated state of the driver (S222).

[0102] If it is determined that the current road segment where automated driving is permitted will soon end (S220: Yes), the detection reference values ​​A are selected to detect an inattentive state of the driver (S212). The reason why the detection reference values ​​C are not selected to detect an agitated state of the driver, despite the fact that an override was performed by the driver during automated driving, is as follows.

[0103] First, in a situation where the current automated driving permission period is about to end, it is conceivable that the driver is aware of this and believes it is necessary to switch to manual driving soon. In such a situation, if the driver knows that overriding the automated driving permission will result in a switch from automated to manual driving, they might switch to manual driving by overriding the permission without following the normal procedure for transitioning from automated to manual driving.

[0104] Or there may be cases where, after entering an inattentive state during automated driving, the driver does not recognize an impending end of the automated driving permission section early enough and hastily operates the steering wheel or presses the brake pedal.

[0105] As described above, if an automated driving permission segment is about to end, even if the driver overrides the override during automated driving, it is not always true that the override is due to severe driver stress. Therefore, in the driving assistance processing of this modification, even after detecting an override during automated driving (S216: Yes), if it is determined that the automated driving permission segment is about to end (S220: Yes), the detection reference values ​​A for a manual driving state are selected according to the switch to manual driving resulting from the override.

[0106] After the acquisition reference values ​​A, B, or C have been selected as described above (S212, S214, S222), it is determined (S224) whether the baseline measurements previously acquired in S208 have reached the selected acquisition reference values. If it is determined that any of the acquired baseline measurements have reached a corresponding selected acquisition reference value (S224: Yes), the driver is alerted in a manner corresponding to the acquisition reference value (S226). The manner in which the driver is alerted is the same as described in connection with the embodiment above, and its description is not repeated here.

[0107] If no baseline measurement has reached a corresponding acquisition reference value (S224: No), processing returns to S206 to receive new image data regarding the driver's face and signals relating to the driver's heart rate and blood pressure. Based on the newly received data and signals, new baseline measurements are acquired (S208), and the subsequent processing steps are carried out as described above.

[0108] In the modification described above, even if an override occurs during automated driving, the determination of whether the driver is in an agitated state is not carried out if the override is determined not to be attributable to driver stress. This avoids a situation where, based on an erroneous determination that the driver is agitated, the driver is alerted by a voice message (e.g., "Aren't you irritated?"). This prevents inappropriately upsetting the driver.

[0109] The modification described above represents a case where, if the current road segment where automated driving is permitted is about to end, even after an override occurs during automated driving, the determination of whether the driver is agitated is not performed. Furthermore, in other cases, if an override is detected that cannot be attributed to driver stress, the determination of whether the driver is agitated does not need to be performed.

[0110] For example, if approaching a service area or parking lot while driving on a highway, the driver may override the system before entering the service area or parking lot. In such a case, the determination of whether the driver is agitated is also not required.

[0111] The embodiments and configurations according to the present invention have been described above, but these do not limit the present invention. Other embodiments and configurations are also possible within the scope of the present invention, which can be obtained by suitably combining technical elements disclosed in different embodiments and configurations.

Claims

[1] Driving assistance device (10) mounted in a vehicle (1) to assist a driver during a journey of the vehicle (1) by detecting a state of the driver, wherein the vehicle (1) is between a manual driving mode in which driving operations for acceleration, braking and steering are performed manually by the driver, and an automated driving state in which at least one of the driving operations is carried out automatically, is interchangeable, wherein the driving assistance device (10) comprises: a driver state monitoring unit (120) that detects whether the driver's state is an inattentive state; an alarm unit (140) which, when the driver's inattentive state is detected, alerts the driver; a driving unit (500) operated by the driver to carry out the driving operations; and a driving state change unit (130) which, when the driver operation of the driving operation unit (500) is detected during the automated driving state of the vehicle (1), changes at least one of the driving operations in the automated driving state to the manual driving state, where when at least one of the driving modes in the automated driving state is switched to the manual driving state by the driving state change unit (130), the driver state detection unit (120) detects whether the driver's state is an agitated state; and If the driver's agitated state is detected, the alerting unit (140) will alert the driver. [2] Driving assistance device (10) according to claim 1, wherein when at least one of the driving modes in the automated driving state is switched to the manual driving state by the driving state change unit (130), the driver state detection unit (120) detects for a predetermined period of time whether the driver is in the excited state. [3] Driving assistance device (10) according to claim 1 or 2, wherein when the agitated state of the driver is detected, the alarm unit (140) alerts the driver differently than when the inattentive state of the driver is detected. [4] Driving assistance device (10) according to one of claims 1 to 3, which further comprises: a driving area detection unit (150) that detects whether an automated driving prohibition area, in which the vehicle (1) is not allowed to drive in the automated driving state, is present in a direction of travel of the vehicle (1), where In cases where the automated driving prohibition area is detected, even if at least one of the driving modes in the automated driving state is switched to the manual driving state by the driving state change unit (130), the driver state detection unit (120) does not detect whether the driver is in the excited state, but detects whether the driver is in the inattentive state. [5] Driving assistance method for a vehicle (1) for assisting a driver in driving the vehicle (1) by detecting a state of the driver, wherein the vehicle (1) is switchable between a manual driving mode in which driving operations for acceleration, braking and steering are performed manually by the driver, and an automated driving state in which at least one of the driving operations is carried out automatically, the driver assistance procedure exhibits: a driver state detection process (S122, S224) that detects whether the driver's state is an inattentive state; an alerting process (S124, S226) that alerts the driver when the driver's inattentive state is detected; and a driving state change process (S118, S218) which, when the driver operation of a driving operation unit (500) is detected during the automated driving state of the vehicle (1), changes at least one of the driving operations in the automated driving state to the manual driving state, wherein if at least one of the driving modes in the automated driving state is switched to the manual driving state in the driving state change process (S118, S218), it is detected in the driver state detection process (S122, S224) whether the driver's state is an agitated state; and In the alerting process (S124, S226) the driver is alerted when the driver's agitated state is detected. [6] Driving assistance device (10) mounted in a vehicle (1) to assist a driver during a journey of the vehicle (1) by detecting a state of the driver, wherein the vehicle (1) is between a manual driving mode in which driving operations for acceleration, braking and steering are performed manually by the driver, and an automated driving state in which at least one of the driving operations is carried out automatically, is interchangeable, wherein the driving assistance device (10) comprises: a processor that has a circuit, a memory that stores a computer program, a driver state detection unit (120) which, by means of the processor executing the computer program, detects whether the driver's state is an inattentive state or an agitated state; an alarm unit (140) which, by means of the processor executing the computer program, alerts the driver when the driver's inattentive state is detected; a driving unit (500) comprising a steering wheel operated by the driver to perform the driving operations; and a driving state change unit (130) which, by means of the processor executing the computer program, changes at least one of the driving modes in the automated driving state to the manual driving state when the driver operation of the driving mode unit (500) is detected during the automated driving state of the vehicle (1), where the driver state detection unit (120) by the processor executing the program, detects the inattentive state of the driver in both the manual driving state and the automated driving state, and if at least one of the driving modes in the automated driving state is switched to the manual driving state by the driving state change unit (130), the driver state detection unit (120) does not detect whether the driver's state is an inattentive state, but detects whether the driver's state is an agitated state; and If the driver's agitated state is detected, the alerting unit (140) will alert the driver.

Images