Driving assistance device, driving assistance method, and driving assistance system

Abstract

A driving assistant device comprises a driver state information acquisition unit (11) which acquires driver state information related to the state of a driver including the alertness level of the driver, a passenger state information acquisition unit (12) which acquires passenger state information related to the state of a passenger, and an assistance control unit (13, 13a, 13b) which: when the alertness level of the driver is determined to be below a first threshold on the basis of the driver state information, and when it is determined, on the basis of the passenger state information, to make the passenger take reduction prevention measures, which are associated with reduction in the alertness level of the driver, as a result of determination whether or not to make the passenger take the reduction prevention measures, performs proposal notification in which notification information for making the passenger take the reduction prevention measures is outputted; and does not perform the proposal notification to the passenger when it is determined not to make the passenger take the reduction prevention measures even through the alertness level of the driver is determined to be below the first threshold on the basis of the driver state information.

Description

Driving assistance device, driving assistance method, and driving assistance system

[0001] The present disclosure relates to a driving assistance device, a driving assistance method, and a driving assistance system.

[0002] Conventionally, there is known a technique for notifying a passenger to take measures when a decrease in the driver's level of wakefulness is detected. For example, Patent Document 1 discloses a wakefulness maintaining device that stimulates the driver to maintain wakefulness by retracting the seat belt, and when the driver's drowsiness level is such that an alarm is required, the wakefulness maintaining device stimulates the passenger to retract the seat belt as well, thereby maintaining the driver's wakefulness with the cooperation of the passenger.

[0003] Japanese Patent Application Laid-Open No. 2023-8263

[0004] When the driver's level of alertness is low, a notification to a passenger to respond to the driver's level of alertness is not necessarily effective. For example, when a passenger's level of alertness is low, even if an attempt is made to have the passenger respond to the driver's level of alertness, the passenger's level of alertness may not immediately recover, and the passenger may not be able to respond to the driver's level of alertness promptly. In this case, it may be meaningless to provide a notification to the passenger to respond to the driver's level of alertness.

[0005] The conventional technologies described above do not take such points into consideration, and therefore there is a problem in that unnecessary alerts may be sent to passengers in order to respond to a decrease in the driver's level of alertness.

[0006] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a driving assistance device that notifies passengers to take measures to address a decrease in the driver's level of alertness without providing unnecessary notifications.

[0007] The driving assistance device of the present disclosure includes a driver state information acquisition unit that acquires driver state information regarding the driver's state including the driver's level of alertness; a passenger state information acquisition unit that acquires passenger state information regarding the passenger's state; and an assistance control unit that, when it is determined based on the driver state information that the driver's level of alertness is less than a first threshold, and when it is determined based on the passenger state information whether or not to have the passenger take measures to prevent a decline in the driver's level of alertness and it is determined that the passenger should take measures to prevent a decline, issues a suggestion notification that outputs notification information to the passenger to cause the passenger to take measures to prevent a decline; and when it is determined based on the driver state information that the driver's level of alertness is less than the first threshold but it is determined that the passenger should not take measures to prevent a decline, does not issue a suggestion notification to the passenger.

[0008] According to the present disclosure, it is possible to notify a passenger to take measures to deal with a decrease in the driver's alertness without providing an unnecessary notification.

[0009] 1 is a diagram illustrating an example of the configuration of a driving assistance device according to a first embodiment. FIG. 2 is a diagram for explaining an example of an image of a change in alertness that leads to a driver falling asleep at the wheel. FIG. 3 is a flowchart for explaining the operation of the driving assistance device according to the first embodiment. FIG. 4 is a flowchart for explaining details of the operation of the suggestion notification necessity determination process performed by the driving assistance device in step ST40 of FIG. 3. FIG. 5 is a flowchart for explaining the operation of the driving assistance device in a case where the assistance control unit has a function of outputting assistance result information in the first embodiment. FIG. 6 is a flowchart for explaining the operation of the driving assistance device in a case where the assistance control unit has a function of checking whether or not the passenger has taken a decline prevention measure after issuing a suggestion notification to the passenger in the first embodiment. FIG. 7 is a diagram showing, in a table format, an example of the content of the mode determination conditions referenced by the assistance control unit in a case where the assistance control unit determines, based on the driver's alertness and the passenger's alertness, the mode of the decline prevention measure to be taken for the passenger. 10A and 10B are diagrams illustrating an example of a hardware configuration of a driving assistance system in which one sensor capable of simultaneously detecting a driver and a passenger is connected to a state detection device. 10B is a diagram illustrating an example of a hardware configuration of a driving assistance system according to the first embodiment. 10C is a diagram illustrating an example of a configuration of a driving assistance system according to the first embodiment. 10D is a diagram illustrating an example of a configuration of a driving assistance system according to the second embodiment. 10E is a flowchart illustrating an operation of a driving assistance system according to the first embodiment. 10F is a diagram illustrating an example of a configuration of a driving assistance system according to the third embodiment. 10G is a diagram illustrating an example of a configuration of a driving assistance system according to the third embodiment. 10H is a flowchart illustrating an operation of a driving assistance system according to the first embodiment. 10H is a diagram illustrating an example of a configuration of a driving assistance system according to the third embodiment. 10I is a flowchart illustrating an operation of a driving assistance system according to the first embodiment. 10I is a diagram illustrating an example of a configuration of a driving assistance system according to the third embodiment. 10I is a flowchart illustrating an operation of a driving assistance system according to the third embodiment.

[0010] Embodiments of the present disclosure will be described in detail below with reference to the drawings. Embodiment 1. A driving assistance device according to embodiment 1 performs control to increase the driver's level of alertness (hereinafter referred to as "alertness improvement control") when the driver's level of alertness is reduced. The driving assistance device also provides a notification (hereinafter referred to as "proposal notification") to a passenger to take measures in response to the driver's level of alertness (hereinafter referred to as "decline prevention measures"). However, the driving assistance device determines whether the passenger's condition is such that the effect of the passenger taking the decline prevention measures is expected, in other words, whether the driving assistance device is able to encourage the passenger to increase the driver's level of alertness. If the driving assistance device determines that the passenger's condition is such that the effect of the passenger taking the decline prevention measures is not expected, the driving assistance device does not provide the suggestion notification to the passenger. This prevents a situation in which the notification to the passenger to take measures in response to the driver's level of alertness is meaningless when the driver's level of alertness is reduced. The meaningless suggestion notification is not only ineffective in preventing a decrease in the driver's alertness, but may also cause unnecessary annoyance to passengers. In the first embodiment, the alertness improvement control is assumed to be control that generates some kind of stimulus for the driver, such as sound or vibration. The decrease prevention response is assumed to be some kind of measure to improve the driver's alertness, such as warning the passenger about the driver's decrease in alertness or encouraging the driver to improve their alertness. Details of the driver alertness improvement control and the suggestion notification to the passenger performed by the driving assistance device will be described later with specific examples. In the first embodiment, the passenger refers to vehicle occupants other than the driver, including passengers in the front passenger seat and passengers in the rear seats. The vehicle occupants include, for example, a child in a child car seat.

[0011] FIG. 1 is a diagram illustrating an example configuration of a driving assistance device 10 according to a first embodiment. The driving assistance device 10 is connected to a state detection device 20, a wakefulness enhancement device 41, and a notification device 42. The state detection device 20 is connected to a first sensor 31 and a second sensor 32. The driving assistance device 10, the state detection device 20, the first sensor 31, the second sensor 32, and the stimulation device 40 are mounted on, for example, a vehicle. The state detection device 20 detects the state of the driver based on sensor information (hereinafter referred to as "first sensor information") obtained by the first sensor 31 detecting the driver. The state detection device 20 also detects the state of the passenger based on sensor information (hereinafter referred to as "second sensor information") obtained by the second sensor 32 detecting the passenger. The first sensor 31 and the second sensor 32 are, for example, an imaging device, a biosensor, or a millimeter-wave radar. In the following first embodiment, as an example, the first sensor 31 and the second sensor 32 are imaging devices including a near-infrared camera or a visible light camera.

[0012] The first sensor 31 captures an image of a driver present in the vehicle cabin. The first sensor 31 may be, for example, a sensor commonly used with a so-called "driver monitoring system (DMS)." The first sensor 31 is installed so as to capture an image of an area within the vehicle cabin that includes at least the driver's upper body and the area where the back of the seat on which the driver sits (e.g., an area corresponding to the seat back and the space in front of the headrest). The first sensor 31 is installed, for example, on an instrument panel or the like. The second sensor 32 captures an image of a passenger present in the vehicle cabin. The second sensor 32 is installed so as to capture an image of an area within the vehicle cabin that includes at least the area where the passenger's upper body and the back of the seat on which the passenger sits (e.g., an area corresponding to the seat back and the space in front of the headrest). The second sensor 32 is installed, for example, on the vehicle's overhead console or near the dashboard where a car navigation system or the like is installed.

[0013] The state detection device 20 includes a state detection unit 21. The state detection unit 21 includes a driver state detection unit 211 and a passenger state detection unit 212. The driver state detection unit 211 acquires first sensor information from the first sensor 31, here, a captured image (hereinafter referred to as a "first captured image") of the driver captured by an imaging device serving as the first sensor 31, and detects the driver's state based on the acquired first captured image. The driver state detected by the driver state detection unit 211 includes at least the driver's alertness. The driver state detection unit 211 detects the driver's state, including at least the driver's alertness, by performing known image processing or physiological measurement on the first captured image. For example, the driver state detection unit 211 may detect the driver's head position or facial features based on the first captured image, and detect the driver's alertness based on the driver's head movement (posture loss), the driver's facial expression, the degree of eye opening, the number of blinks, the degree of eye opening, etc. Furthermore, for example, the driver state detection unit 211 may detect the state of the driver, including the driver's level of alertness, using a machine learning trained model (hereinafter referred to as a "machine learning model") that inputs an image of a person and outputs the level of alertness of the person. Note that in the first embodiment, the level of alertness is expressed as a numerical value between 0 and 1.0. The closer the level of alertness is to 1.0, the more alert the driver is.

[0014] The passenger status detection unit 212 acquires second sensor information from the second sensor 32, here an image of a passenger captured by an imaging device serving as the second sensor 32 (hereinafter referred to as a “second captured image”), and detects the presence or absence of a passenger and, if a passenger is present, the state of the passenger based on the acquired second captured image. Here, the state of the passenger detected by the passenger status detection unit 212 includes, for example, the passenger’s alertness. First, the passenger status detection unit 212 performs known image processing based on the second captured image to detect the presence or absence of a passenger. For example, the passenger status detection unit 212 may detect the presence or absence of a passenger using a template matching technique, or may detect the presence or absence of a passenger using a machine learning model that inputs the captured image and outputs information indicating the presence or absence of a person in the captured image. If the passenger status detection unit 212 detects the presence of a passenger, it detects the state of the passenger, here, the alertness level, by performing known image processing or physiological measurement on the second captured image. The passenger state detection unit 212 may detect the level of alertness of the passenger in the same manner as the driver state detection unit 211 detects the level of alertness of the driver.

[0015] The state detection unit 21 outputs driver state information indicating the state of the driver detected by the driver state detection unit 211 and passenger state information indicating the state of the passenger detected by the passenger state detection unit 212 to the driving assistance device 10. The driver state information includes, for example, information indicating the driver's alertness. The passenger state information includes information indicating the presence or absence of a passenger and the state of the passenger, in this case, information indicating the alertness of the passenger.

[0016] Note that, here, it is assumed that the function of detecting the driver's state and the function of detecting the passenger's state are provided in one device, in other words, in the state detection device 20, but this is merely an example. For example, the function of detecting the driver's state and the function of detecting the passenger's state may be provided in separate devices. For example, a driver state detection device having the function of the driver state detection unit 211 and a passenger state detection device having the function of the passenger state detection unit 212 may be provided separately. Furthermore, the function of the driver state detection unit 211 may be provided in the first sensor 31, and the function of the passenger state detection unit 212 may be provided in the second sensor 32. In this case, for example, the first sensor 31 is the driver state detection device, and the second sensor 32 is the passenger state detection device.

[0017] The driving assistance device 10 acquires driver status information and passenger status information from the status detection device 20, and performs driver alertness enhancement control or provides a suggestion notification to the passenger based on the acquired driver status information and passenger status information. Specifically, when the driving assistance device 10 determines that the driver's alertness is reduced based on the driver status information acquired from the status detection device 20, the driving assistance device 10 outputs information for performing alertness enhancement control (hereinafter referred to as "alertness enhancement control information") to the alertness enhancement device 41. The alertness enhancement device 41 is a device for enhancing the driver's alertness and is a device that generates audio stimuli, tactile stimuli, or the like. Examples of the alertness enhancement device 41 include an audio output device provided in a car navigation system or the like that outputs sound or voice, a vibration generator provided in the driver's seat that generates vibrations in the driver's seat, a vibration generator provided in the steering wheel that generates vibrations in the steering wheel, or a seatbelt pretensioner that controls the retraction of a seatbelt in the driver's seat. 1 illustrates only one alertness enhancement device 41, but this is merely an example, and multiple alertness enhancement devices 41 may be connected to the driving assistance device 10. Multiple different types of alertness enhancement devices 41, such as a sound output device and a vibration generating device, may be connected to the driving assistance device 10. The driving assistance device 10 outputs alertness enhancement control information according to the type of alertness enhancement device 41.

[0018] Furthermore, when the driving assistance device 10 determines that the driver's alertness level has decreased, it determines whether to have the passenger take a countermeasure to prevent a decline in alertness, based on the passenger's state (here, the alertness level). If it determines that the passenger should take a countermeasure to prevent a decline in alertness, it issues a notification suggestion to the passenger. Specifically, the driving assistance device 10 outputs information for causing the passenger to take a countermeasure to prevent a decline (hereinafter referred to as "notification information") to the notification device 42. The notification device 42 is a device for causing the passenger to take a countermeasure to prevent a decline in alertness. For example, the notification device 42 may be a display device that displays a message or an audio output device, such as a car navigation system. For example, the notification device 42 may be a vibration generating device that generates vibrations in the seat in which the passenger is seated, or a seatbelt pretensioner that controls the retraction of the seatbelt in which the passenger is seated. The driving assistance device 10 prompts the passenger to take measures to prevent the vehicle from falling by causing the notification device 42 to display, output audio, vibrate, or retract the seat belt. Although only one notification device 42 is shown in FIG. 1 , this is merely an example, and multiple notification devices 42 may be connected to the driving assistance device 10. Multiple different types of notification devices 42, such as an audio output device and a vibration generating device, may be connected to the driving assistance device 10. The driving assistance device 10 outputs notification information according to the type of notification device 42.

[0019] A description will be given of an example of the configuration of the driving assistance device 10. As shown in FIG. 1, the driving assistance device 10 includes a driver state information acquisition unit 11, a passenger state information acquisition unit 12, and an assistance control unit 13.

[0020] The driver state information acquisition unit 11 acquires driver state information from the driver state detection unit 211 of the state detection device 20. The driver state information acquisition unit 11 outputs the acquired driver state information to the assistance control unit 13.

[0021] The passenger status information acquisition unit 12 acquires passenger status information from the passenger status detection unit 212 of the status detection device 20. The passenger status information acquisition unit 12 outputs the acquired passenger status information to the assistance control unit 13.

[0022] The assistance control unit 13 determines whether the driver's alertness is reduced based on the driver state information acquired by the driver state information acquisition unit 11. Specifically, the assistance control unit 13 determines whether the driver's alertness is less than a preset threshold (hereinafter referred to as a "first threshold") based on the driver state information. The first threshold is set in advance by a developer or the like and stored in an internal buffer or the like of the assistance control unit 13. The developer or the like sets the first threshold to, for example, "0.8," which corresponds to an alertness level between an alertness level estimated as a fully alert state and an alertness level estimated as a slightly reduced state.

[0023] FIG. 2 is a diagram illustrating an example of how a driver's alertness changes as they fall asleep at the wheel. The alertness level will be described with reference to FIG. 2. In FIG. 2, the vertical axis represents alertness, and the horizontal axis represents the time elapsed since the start of driving. Generally, a driver's alertness is high when the driver starts driving and tends to decrease over time. FIG. 2 classifies alertness into a plurality of alertness levels (DL1 to DL5), and illustrates an example of a typical situation that a driver may encounter over time after starting driving. The driver's alertness gradually decreases over time, fluctuating due to the influence of the surroundings of the vehicle, the interior of the vehicle, and the driver's own psychology, physiology, or physical condition. The driver's alertness gradually decreases, resulting in poor posture, frequent eye closure, and microsleep, progressing from a light state of drowsiness to a full state of drowsiness.

[0024] The alertness level "DL5" refers to a state estimated to be fully alert. In this case, the driver is estimated to be able to blink quickly, have wide eye opening, and maintain a stable posture. The alertness level "DL4" refers to a state estimated to be slightly reduced alertness. In this case, the driver is estimated to be blinking with long eye closure periods, or with a slightly narrower eye opening. The alertness level "DL3" refers to a state estimated to be moderately reduced alertness. In this case, the driver is estimated to be blinking with long eye closure periods, with a narrower eye opening, or, in some cases, yawning or changes in facial expression. The driver's driving operations are also estimated to be somewhat sluggish. The alertness level "DL2" refers to a state estimated to be strongly reduced alertness. In this case, the driver is estimated to be repeatedly losing posture and recovering, blinking frequently with long eye closure periods, closing the eyes frequently for long periods of time such as three seconds, or experiencing microsleep. It is also estimated that driving will be impaired. The alertness level DL2 is a level at which the driver's alertness needs to be restored, i.e., the driver's alertness needs to be improved. The alertness level "DL1" refers to a state estimated to be a mild state of drowsiness. In this case, the driver is estimated to have poor posture (posture change) or have their eyes completely closed. When the alertness level is "DL1," the driver can respond to warnings but is unable to drive. The alertness level "DL0" refers to a state estimated to be a complete state of drowsiness. In this case, the driver is estimated to have a persistent poor posture or have their eyes completely closed. When the alertness level is "DL0," the driver cannot respond to warnings. While it is not expected that the driver's alertness will decrease to "DL0" during driving, it is possible that the driver's alertness level may decrease to "DL0" due to, for example, severe fatigue or the effects of alcohol consumption. When the alertness level is "DL0", the driver is in a state where he or she is unable to drive.

[0025] In the first embodiment, as shown in FIG. 2 , a wakefulness level of 0.8 to 1.0 corresponds to a wakefulness level "DL5," a wakefulness level of 0.6 to 0.8 corresponds to a wakefulness level "DL4," a wakefulness level of 0.5 to 0.6 corresponds to a wakefulness level "DL3," a wakefulness level of 0.3 to 0.5 corresponds to a wakefulness level "DL2," a wakefulness level of 0.2 to 0.3 corresponds to a wakefulness level "DL1," and a wakefulness level of 0 to 0.2 corresponds to a wakefulness level "DL0." The developer or the like may set the first threshold to, for example, "0.8," which is between the wakefulness levels "DL5" and "DL4." Note that this is merely an example, and the first threshold may be set to a value other than "0.8," such as "0.6." The above-described wakefulness level division is merely an example, and a level division other than that shown in FIG. 2 may be used depending on the wakefulness level. In the above description, an example of the change in the driver's alertness that leads to drowsiness at the wheel has been described using FIG. 2, but the alertness of a passenger generally changes in the same way as the driver.

[0026] If the driver's alertness is less than a first threshold based on the driver state information, the assistance control unit 13 determines that the driver's alertness is reduced. If the assistance control unit 13 determines that the driver's alertness is reduced, i.e., if the assistance control unit 13 determines that the driver's alertness is less than the first threshold, the assistance control unit 13 outputs alertness improvement control information to the alertness improvement device 41 for performing alertness improvement control on the driver. Based on the alertness improvement control information output from the assistance control unit 13, the alertness improvement device 41 outputs, for example, a warning sound or a voice message alerting the driver to the fact that drowsiness is reduced. For example, the alertness improvement device 41 vibrates the seat or steering wheel based on the alertness improvement control information. For example, the alertness improvement device 41 tightens the tension on the seat belt. In this way, the alertness improvement device 41 provides the driver with a stimulus to increase their alertness.

[0027] The assistance control information only needs to be information for performing some kind of control to improve the driver's level of alertness, and it is sufficient that the type of control to improve the driver's level of alertness that the assistance control information is to perform is determined in advance.

[0028] Furthermore, when the assistance control unit 13 determines that the driver's alertness has decreased, i.e., when the assistance control unit 13 determines that the driver's alertness is less than the first threshold, the assistance control unit 13 determines the state of the passenger based on the passenger state information, and determines whether or not to have the passenger take measures to prevent a decline in their driving ability, depending on the determined state of the passenger. When the assistance control unit 13 determines that a decline prevention measure should be taken, the assistance control unit 13 outputs notification information to the notification device 42 to have the passenger take measures to prevent a decline in their driving ability.

[0029] Here, an example of a method by which the assistance control unit 13 determines whether to have the passenger take a decrease prevention measure based on the passenger state information will be described. In the first embodiment, the passenger's state (hereinafter referred to as the "measure availability determination state") is used to determine whether to have the passenger take a decrease prevention measure, in other words, whether the passenger is in a state where they can take action to improve the driver's wakefulness. In the first embodiment, the assistance control unit 13 determines whether to have the passenger take a decrease prevention measure based on, for example, whether the passenger's state based on the passenger state information, in this case, the wakefulness, satisfies a response availability condition. If the passenger's wakefulness meets the response availability condition, the assistance control unit 13 causes the passenger to take a decrease prevention measure, in other words, determines that the passenger is in a state where they can take action to improve the driver's wakefulness. On the other hand, if the passenger's wakefulness does not meet the response availability condition, the assistance control unit 13 does not cause the passenger to take a decrease prevention measure, in other words, determines that the passenger is not in a state where they can take action to improve the driver's wakefulness. Since the passenger status information includes information on the passenger's alertness, the assistance control unit 13 can make the above-mentioned determination based on the passenger status information. Note that the assistance control unit 13 makes the above-mentioned determination when a passenger is present. The assistance control unit 13 can determine whether a passenger is present from the passenger status information.

[0030] The response possible conditions are conditions that define a state in which it is estimated that a passenger can take action to improve the driver's alertness, and are set in advance by developers, etc. When the developers, etc. set the response possible conditions, they store information indicating the response possible conditions (hereinafter referred to as "response possible condition information") in a buffer or the like inside the assistance control unit 13.

[0031] Here, the possible response conditions are set, for example, as the following <possible response condition (1)>: <possible response condition (1)> The passenger's alertness level is equal to or higher than the second threshold.

[0032] The second threshold is set in advance by a developer or the like together with the response condition and stored in an internal buffer or the like of the assistance control unit 13. When the passenger's alertness level based on the passenger status information satisfies the response condition (1), that is, when the passenger's alertness level is equal to or higher than the second threshold, the assistance control unit 13 determines to take a response to prevent a decline in the passenger's alertness. As described above, the passenger status information includes information indicating the passenger's alertness level.

[0033] The assistance control unit 13 then outputs notification information to the notification device 42. The notification information is, for example, information that causes an audio output of a message urging the driver to pay attention because the driver's level of alertness has decreased. When the notification information is output from the assistance control unit 13, the notification device 42 outputs or displays an audio message such as, "The driver seems to be getting drowsy. Please make sure the driver pays attention." This allows the passenger to understand that the driver's level of alertness has decreased and that some measure should be taken to improve the driver's level of alertness.

[0034] The content of the notification information described above is merely an example, and the notification information may be, for example, information that displays a message urging the driver to pay attention because the driver's level of alertness is low. The notification information may also be, for example, information that causes a voice output or display of a message urging the passenger to talk to the driver. For example, the assistance control unit 13 may output notification information that causes the notification device 42 to output a voice message such as, "The driver seems sleepy. Why don't you talk to him?" The assistance control unit 13 thereby urges the passenger to talk to the driver.

[0035] The notification information only needs to be information for encouraging the passenger to take some kind of measure to improve the driver's alertness, and it may be predetermined as to what kind of measure to take to improve the driver's alertness, i.e., what kind of response to prevent deterioration, the notification information is intended to encourage the passenger to take. Note that some kind of response to improve the driver's alertness that is encouraged by the notification information is a response to prevent deterioration. In the above example, for example, talking to the driver to improve the driver's alertness is one example of a response to prevent deterioration.

[0036] For example, the assistance control unit 13 may provide a notification suggesting different measures to be taken by the passenger to prevent a decline depending on the driver's alertness. For example, when the driver's alertness is between "0.6" and "0.8," which corresponds to an alertness level "DL4," the assistance control unit 13 may output a voice message or display notification information saying, "The driver seems a little sleepy. Please make him / her take more care." When the driver's alertness is between "0.5" and "0.6," which corresponds to an alertness level "DL3," the assistance control unit 13 may output a voice message or display notification information saying, "The driver seems sleepy. Why don't you talk to him / her?" When the driver's alertness is between "0.3" and "0.5," which corresponds to an alertness level "DL2," the assistance control unit 13 may output a voice message or display notification information saying, "The driver seems very sleepy. Some action is needed. Why don't you give him / her some stimulation?"

[0037] The assistance control unit 13 will not make a suggestion notification to the passenger if there is no passenger present, or if there is a passenger present but the passenger's level of alertness does not satisfy the above-mentioned <Response Condition (1)>.

[0038] The operation of the driving assistance device 10 according to the first embodiment will be described. FIG. 3 is a flowchart for explaining the operation of the driving assistance device 10 according to the first embodiment. Note that the operation of the driving assistance device 10 will be described here assuming that a condition such as <supportable condition (1)> is set as the supportable condition. The driving assistance device 10 performs the operation shown in the flowchart of FIG. 3 at a predetermined timing, such as a one-minute cycle. For example, the state detection device 20 may measure the wakefulness levels of the driver and passenger at a predetermined timing, such as a one-minute cycle, and the driving assistance device 10 may perform the operation shown in the flowchart of FIG. 3 every time the state detection device 20 outputs driver state information and passenger state information.

[0039] The driver state information acquisition unit 11 acquires the driver state information from the driver state detection unit 211 of the state detection device 20 (step ST10). The driver state information acquisition unit 11 outputs the acquired driver state information to the assistance control unit 13.

[0040] The passenger status information acquisition unit 12 acquires passenger status information from the passenger status detection unit 212 of the status detection device 20 (step ST20). The passenger status information acquisition unit 12 outputs the acquired passenger status information to the assistance control unit 13.

[0041] The assistance control unit 13 determines whether the driver's alertness is reduced based on the driver state information acquired in step ST10 by the driver state information acquisition unit 11. Specifically, the assistance control unit 13 determines whether the driver's alertness is less than a first threshold based on the driver state information (step ST30).

[0042] In step ST30, if the driver's level of alertness is less than the first threshold ("YES" in step ST30), the assistance control unit 13 determines that the driver's level of alertness has decreased. Then, the assistance control unit 13 determines the state of the passenger based on the passenger state information, and performs a suggestion notification necessity determination process (step ST40) to determine whether or not to have the passenger take measures to prevent a decrease in the driver's level of alertness, depending on the determined state of the passenger.

[0043] If it is determined in step ST40 that the passenger should take measures to prevent the driver from falling (YES in step ST50), the assistance control unit 13 outputs, to the alertness enhancement device 41, alertness enhancement control information for controlling the driver to increase his / her alertness (step ST60).The assistance control unit 13 also outputs, to the alarm device 42, alert information for causing the passenger to take measures to prevent the driver from falling (step ST70).

[0044] On the other hand, if it is determined in step ST40 that the passenger should not be prompted to take measures to prevent a decline (if "NO" in step ST50), the assistance control unit 13 outputs alertness improvement control information for performing alertness improvement control on the driver (step ST80). The assistance control unit 13 does not output alert information to the notification device 42.

[0045] In step ST30, if the driver's wakefulness is equal to or greater than the first threshold value ("NO" in step ST30), the driving assistance device 10 does not perform the processes of steps ST40 to ST80.

[0046] In the operation of the driving assistance device 10 shown in the flowchart of Fig. 3, the processing is performed in the order of step ST10 and step ST20, but this is merely an example, and for example, the processing may be performed in the order of step ST20 and step ST10, or the processing of step ST10 and the processing of step ST20 may be performed in parallel. Also, in the operation of the driving assistance device 10 shown in the flowchart of Fig. 3, the processing is performed in the order of step ST60 and step ST70, but this is merely an example, and for example, the processing may be performed in the order of step ST70 and step ST60, or the processing of step ST60 and the processing of step ST70 may be performed in parallel. 3, the processes of steps ST60 and ST80 are performed after the process of step ST50, but this is merely an example, and for example, if it is determined in step ST30 that the driver's alertness is less than the first threshold, the assistance control unit 13 may output alertness improvement control information to the alertness improvement device 41 before the process of step ST40 is performed. In this case, the processes of steps ST60 and ST80, which are performed after the process of step ST50, are not necessary.

[0047] FIG. 4 is a flowchart for explaining details of the operation of the suggestion notification necessity determination process performed by the driving assistance device 10 in step ST40 of FIG.

[0048] The assistance control unit 13 determines whether or not a fellow passenger is present based on the fellow passenger status information (step ST1001).

[0049] If it is determined in step ST1001 that a passenger is present ("YES" in step ST1001), the assistance control unit 13 determines whether the passenger's state based on the passenger state information, in this case, the passenger's alertness, satisfies the response condition (step ST1002). If it is determined in step ST1002 that the passenger's alertness satisfies the response condition, in this case, the passenger's alertness is equal to or greater than the second threshold ("YES" in step ST1002), the assistance control unit 13 determines to have the passenger take a response to prevent a decline (step ST1003).

[0050] If it is determined in step ST1001 that there is no passenger present (if the answer is "NO" in step ST1001), or if it is determined in step ST1002 that the passenger's level of alertness does not satisfy the response condition, in this case, that the passenger's level of alertness is less than the second threshold (if the answer is "NO" in step ST1002), the assistance control unit 13 determines not to have the passenger take response to prevent a decline (step ST1004).

[0051] In this way, when the driving assistance device 10 determines that the driver's alertness is less than the first threshold based on the driver state information and that the passenger's state based on the passenger state information (here, the passenger's alertness) satisfies the response condition, the driving assistance device 10 determines that the passenger should take a response to prevent a decline and provides a notification suggesting the decline. On the other hand, when the driving assistance device 10 determines that the driver's alertness is less than the first threshold based on the driver state information but the passenger's alertness does not satisfy the response condition, the driving assistance device 10 determines that the passenger should not take a response to prevent a decline and does not provide a notification suggesting the decline. This prevents a situation in which providing a notification to the passenger to respond to the decline in the driver's alertness when the driver's alertness is declining is meaningless. As a result, the driving assistance device 10 prevents the passenger from being unnecessarily annoyed. When the driver's level of alertness decreases, the driving assistance device 10 makes a suggestion notification to the passenger if the passenger's level of alertness is equal to or higher than a second threshold and the passenger is in a state where he or she can take action against the driver, and does not make an unnecessary suggestion notification to the passenger if the passenger's level of alertness is less than the second threshold and the passenger is in a state where he or she cannot take action against the driver.

[0052] <Modification (1-1)> In the above-described first embodiment, for example, when the assistance control unit 13 has notified the passenger of a suggestion, the assistance control unit 13 may detect, based on the driver state information, a change in the driver's alertness before and after the suggestion is notified to the passenger, and output information related to the detected change in the driver's alertness (hereinafter referred to as "assistance result information") to the notification device 42. The assistance result information is, for example, information that causes a message explaining whether or not there has been a change in the driver's alertness to be output by voice or displayed.

[0053] For example, when the assistance control unit 13 notifies the passenger of a suggestion, it sets an assistance completion flag provided in an internal buffer or the like of the assistance control unit 13 to "1" and stores the driver's alertness based on the driver state information in the internal buffer or the like. The initial value of the assistance completion flag is "0" and the assistance completion flag is initialized when the driver's alertness becomes equal to or greater than a first threshold, when the vehicle power is turned off, or the like. After setting the assistance completion flag to "1", the assistance control unit 13 compares the driver's alertness based on the acquired driver state information with the driver's alertness stored in the internal buffer or the like to detect a change in the driver's alertness. Then, the assistance control unit 13 outputs assistance result information to the notification device 42.

[0054] For example, when the assistance control unit 13 sets the assistance completion flag to “1,” the assistance control unit 13 detects a change in the driver's alertness as described above after a certain time (hereinafter referred to as the “awakening level change detection time”) has elapsed since the assistance completion flag was set to “1.” For example, when the assistance control unit 13 sets the assistance completion flag to “1,” the assistance control unit 13 determines whether the alertness change detection time has elapsed by counting up a timer provided in an internal buffer or the like. The alertness change detection time is, for example, five minutes. The alertness change detection time is set in advance by a developer or the like, and information indicating the alertness change detection time (hereinafter referred to as “awakening level change detection time information”) is stored in an internal buffer or the like of the assistance control unit 13. When the alertness change detection time has elapsed, the assistance control unit 13 acquires driver status information from the driver status detection unit 211 of the status detection device 20 and compares the driver's alertness based on the acquired driver status information with the driver's alertness stored in an internal buffer or the like, as described above. The assistance control unit 13 may acquire the driver state information directly from the driver state detection unit 211 or via the driver state information acquisition unit 11 .

[0055] For example, the assistance control unit 13 may detect a change in the driver's alertness before and after the suggestion notification is made to the passenger, and output assistance result information if the driver's alertness has not been improved. In this case, if the driver's alertness has not been improved, the assistance control unit 13 outputs assistance result information for audio output or display of a message informing the driver that the driver's alertness has not been improved, such as "The driver's alertness has not been improved." If the driver's alertness has been improved, the assistance control unit 13 does not output the assistance result information. Note that the assistance control unit 13 compares the driver's alertness based on the acquired driver state information with the driver's alertness stored in an internal buffer or the like, and the extent to which the change in the alertness is required to determine that the driver's alertness has been improved is predetermined.

[0056] For example, the assistance control unit 13 may repeatedly determine whether to output assistance result information every time the alertness change detection time elapses, as described above, and when it is determined that assistance result information should be output multiple times, the content of the message or the like to be output in the assistance result information may be changed between the current time and the previous time. For example, after previously outputting assistance result information that outputs or displays a message saying, "The driver's alertness has not improved," if it is determined again this time that the driver's alertness has not improved, the assistance control unit 13 may output assistance result information that outputs or displays a message saying, "The driver seems to be feeling sleepy. Perhaps you should take a rest somewhere," or "The driver seems to be feeling sleepy. Perhaps you should play music, or perhaps you should chew gum, etc."

[0057] Fig. 5 is a flowchart for explaining the operation of the driving assistance device 10 in a case where the assistance control unit 13 has a function of outputting assistance result information in Embodiment 1. In Fig. 5, the details of the processing of steps ST10 to ST80 are the same as the details of the processing of steps ST10 to ST80 described using the flowchart in Fig. 3, and therefore, redundant description will be omitted.

[0058] In step ST70, the assistance control unit 13 outputs notification information to the notification device 42 to prompt the passenger to take measures to prevent a decline in alertness, and then starts counting the alertness change detection time. The assistance control unit 13 then determines whether the alertness change detection time has elapsed (step ST90). If the alertness change detection time has not elapsed (if "NO" in step ST90), the assistance control unit 13 repeats the process of step ST90 until the alertness change detection time has elapsed. If the alertness change detection time has elapsed (if "YES" in step ST90), the assistance control unit 13 acquires driver state information from the driver state detection unit 211 and detects a change in the driver's alertness before and after the suggestion notification to the passenger based on the acquired driver state information (step ST91). The assistance control unit 13 then outputs assistance result information to the notification device 42 (step ST92). If the driver's alertness has not improved in step ST92, the assistance control unit 13 may output the assistance result information.

[0059] For example, if the assistance control unit 13 repeatedly determines whether to output assistance result information each time the time for detecting a change in alertness has elapsed, the assistance control unit 13 clears the timer that counts the time for detecting a change in alertness after processing step ST92, and performs the processing of steps ST90 to ST92 again. In this case, if the assistance control unit 13 determines in step ST92 that assistance result information should be output, the content of the message or the like to be output in the assistance result information may be changed from the previous time. For example, the assistance control unit 13 repeats the processing of steps ST90 to ST92 until a preset upper limit number of times is reached.

[0060] In this way, since the assistance control unit 13 has the function of outputting assistance result information, the driving assistance device 10 can allow the passenger to know whether the driver's alertness has changed as a result of the deterioration prevention measures. Furthermore, by outputting assistance result information when the driver's alertness has not improved, the driving assistance device 10 can inform the passenger whether the deterioration prevention measures have been effective, or can suggest other deterioration prevention measures if they are estimated to be ineffective.

[0061] <Modification (1-2)> In the above-described first embodiment, the assistance control unit 13 may have a function of confirming whether or not the passenger has taken measures to prevent a decline after issuing a suggestion notification to the passenger. More specifically, in the above-described first embodiment, after issuing a suggestion notification to the passenger, the assistance control unit 13 may determine, based on the passenger status information, whether or not the passenger has taken measures to prevent a decline based on the suggestion notification, and if it is determined that the passenger has not taken measures to prevent a decline, the assistance control unit 13 may issue a suggestion notification to the passenger again.

[0062] For example, after outputting the notification information to the passenger, the assistance control unit 13 acquires passenger status information from the passenger status detection unit 212 of the status detection device 20 and determines whether the passenger has taken measures to prevent a decline, as described above, based on the passenger status information. The assistance control unit 13 may acquire the passenger status information directly from the passenger status detection unit 212 or via the passenger status information acquisition unit 12. In this case, the passenger status detected by the passenger status detection unit 212 includes the passenger's behavior. The passenger status detection unit 212 can detect the passenger's behavior, for example, by performing known image processing on the second captured image. The passenger's behavior includes, for example, the passenger's hand movement, facial movement, or mouth movement. The passenger status detection unit 212 may detect the passenger's speech as the passenger's behavior, for example, by recognizing the passenger's mouth movement using a known lip reading technique based on the second captured image. In addition, the passenger status detection unit 212 may obtain voice recognition results from, for example, a voice recognition device (not shown) connected to the passenger status detection unit 212 that recognizes the content of speech within the vehicle cabin, and detect the content of the passenger's speech as the passenger's behavior.

[0063] When the assistance control unit 13 determines that the passenger is making some kind of request to the driver based on the passenger status information acquired from the passenger status detection unit 212, the assistance control unit 13 determines that the passenger has taken a decrease prevention measure. The assistance control unit 13 may determine whether the passenger is making some kind of request to the driver in accordance with, for example, a behavior determination condition. The behavior determination condition defines what behavior of the passenger determines that the passenger is making some kind of request to the driver, in other words, that the passenger has taken a decrease prevention measure. The behavior determination condition is set in advance by a developer or the like, and information indicating the behavior determination condition (hereinafter referred to as "behavior determination condition information") is stored in a buffer or the like inside the assistance control unit 13. The behavior determination condition may be set, for example, such that the passenger is determined to be making some kind of request to the driver if the passenger is touching the driver with a hand or the like, turning their face toward the driver, or talking to the driver. When the assistance control unit 13 determines based on the passenger state information that the passenger has not taken any action to the driver, it determines that the passenger has not taken any measures to prevent a decline.

[0064] The assistance control unit 13 may determine whether the passenger has taken measures to prevent a decline, for example, based on a combination of the passenger status information and the driver status information. That is, the behavior determination condition may define what behavior (movement or conversation) the passenger is engaging in and what behavior (movement or conversation) the driver is engaging in, based on which it is determined that the passenger has engaged in some kind of interaction with the driver. In this case, when the assistance control unit 13 outputs the notification information to the passenger, it acquires passenger status information from the passenger status detection unit 212 of the status detection device 20 and driver status information from the driver status detection unit 211 of the status detection device 20. Based on the passenger status information and the driver status information, the assistance control unit 13 determines whether the passenger has engaged in some kind of interaction with the driver, based on the behaviors of both the passenger and the driver.

[0065] If the assistance control unit 13 determines that the passenger has not taken any measures to prevent the vehicle from falling, in other words, that the passenger has not taken any action to encourage the driver, the assistance control unit 13 again issues a suggestion notification to the passenger. More specifically, the assistance control unit 13 again outputs notification information to the notification device 42.

[0066] 6 is a flowchart for explaining the operation of the driving assistance device 10 in a case where the assistance control unit 13 has a function of checking whether the passenger has taken measures to prevent a decline after issuing a suggestion notification to the passenger in Embodiment 1. In FIG. 6, the details of the processes of steps ST10 to ST80 are the same as the details of the processes of steps ST10 to ST80 described using the flowchart in FIG. 3, and therefore, redundant explanations will be omitted.

[0067] In step ST70, the assistance control unit 13 outputs, to the notification device 42, notification information for instructing the passenger to take measures to prevent the passenger from lowering, and then acquires passenger status information from the passenger status detection unit 212, and determines, based on the acquired passenger status information, whether the passenger has taken measures to prevent the passenger from lowering, in other words, whether the passenger has made some kind of request to the driver (step ST100). When determining, based on the combination of the passenger status information and the driver status information, whether the passenger has taken measures to prevent the passenger from lowering, the assistance control unit 13 acquires, in step ST100, the passenger status information from the passenger status detection unit 212 and the driver status information from the driver status detection unit 211, and determines, based on the acquired passenger status information and driver status information, whether the passenger has taken measures to prevent the passenger from lowering, in other words, whether the passenger has made some kind of request to the driver. If it is determined in step ST100 that the passenger has not taken measures to prevent the vehicle from lowering ("NO" in step ST100), the assistance control unit 13 outputs notification information to the notification device 42 (step ST101). If it is determined in step ST100 that the passenger has taken measures to prevent the vehicle from lowering ("YES" in step ST100), the operation of the driving assistance device 10 does not perform the process of step ST101.

[0068] In this way, since the assistance control unit 13 has the function of checking whether the passenger has taken measures to prevent a decline after making a suggestion notification to the passenger, the driving assistance device 10 can more reliably ensure that the passenger takes measures to prevent a decline, and cooperate in improving the driver's level of alertness.

[0069] The assistance control unit 13 may combine the content of the above-described <Modification (1-1)> with the content of the <Modification (1-2)>. For example, when the assistance control unit 13 outputs notification information to the passenger, in other words, when the assistance control unit 13 determines that the passenger has made some kind of intervention toward the driver after making a suggestion notification to the passenger, the assistance control unit 13 may detect a subsequent change in the driver's alertness and output assistance result information to the notification device 42. For example, the assistance control unit 13 may detect a change in the driver's alertness before and after making the suggestion notification to the passenger, and, if the driver's alertness is improved, output the assistance result information for audio output or display of a message indicating that the driver's alertness has been improved, such as "The driver is recovering." If the driver's alertness is not improved, output the assistance result information for audio output or display of a message indicating that the driver's alertness has not been improved, such as "The driver does not seem to be recovering." In this case, in the operation of the driving assistance device 10 described using the flowchart of Figure 6, if the assistance control unit 13 determines, for example, in step ST100 that the passenger has taken measures to prevent a decline (if "YES" in step ST100), it performs processing of steps ST90 to ST92 in the flowchart of Figure 5.

[0070] <Modification (1-3)> In the first embodiment described above, the passenger's response capability determination state is determined based on the passenger's alertness. In other words, whether the passenger is in a state where they can take action to improve the driver's alertness is determined based on the passenger's alertness. However, this is merely an example, and the passenger's response capability determination state may be a state other than the alertness. For example, the passenger's response capability determination state may be determined based on the passenger's attributes. In the first embodiment, the "attribute" refers to a person attribute that represents an individual person's characteristics (such as age, developmental stage, health condition, or medical condition), such as adult, infant, toddler, or person with dementia. In this case, the response capability condition is set to, for example, a condition such as the following <Response capability condition (2)>. <Response capability condition (2)> The passenger is not a non-target passenger defined in the non-target passenger condition.

[0071] For example, the developer or the like may define passengers who are assumed to be in a state in which they are not capable of taking measures to prevent a decline as non-target passengers, set conditions indicating the attributes of passengers who are non-target passengers (hereinafter referred to as "non-target passenger conditions") together with the response conditions, and store the response condition information indicating the response conditions and non-target passenger conditions in a buffer or the like inside the assistance control unit 13. For example, the non-target passenger conditions may be set as "babies, young children, or people with dementia."

[0072] Based on the passenger status information, if the passenger's status, in this case, attributes, satisfies the above-mentioned <Response condition (2)>, i.e., if the passenger is not a non-target passenger, the assistance control unit 13 determines that a decline prevention response should be taken for the passenger.

[0073] In this case, the state of the passenger detected by the passenger state detection unit 212 includes the passenger's attributes. The passenger state detection unit 212 can detect the passenger's attributes by, for example, performing known image processing on the second captured image. The passenger state detection unit 212 may, for example, detect the passenger's speech content by recognizing the passenger's mouth movements using a known lip reading technique based on the second captured image, and detect the passenger's attributes from the detected passenger's speech content. Alternatively, the passenger state detection unit 212 may, for example, obtain a speech recognition result from a speech recognition device connected to the passenger state detection unit 212, detect the passenger's speech content, and detect the passenger's attributes from the detected passenger's speech content. Alternatively, the passenger state detection unit 212 may detect the passenger's attributes based on, for example, whether an ID transmission device, such as a smartphone, owned by the passenger is detected. For example, when a passenger is present but an ID transmission device is not detected, passenger status detection unit 212 detects that the passenger is an infant, a small child, or a person with dementia. Passenger status detection unit 212 may also acquire load information from a load sensor (not shown) connected to passenger status detection unit 212 that measures the load on the seat, and detect the attributes of the passenger from the load information. Passenger status detection unit 212 may also acquire passenger attribute information indicating the attributes of the passenger that have been input by the driver or passenger and accepted by an attribute acceptance device (not shown) connected to passenger status detection unit 212, and detect the attributes of the passenger from the passenger attribute information. In addition, the passenger status detection unit 212 may, for example, perform known image processing on the second captured image to detect the passenger's physique, skeleton, or skeletal movement, and detect the passenger's attributes, such as whether the passenger is an adult, infant, or toddler, from the detected passenger's physique, skeleton, or skeletal movement.

[0074] Here, the assistance control unit 13 determines whether or not the passenger is a non-target passenger based on the passenger status information and the response conditions, but this is just one example. For example, when the passenger status detection unit 212 detects the attributes of a passenger, it determines whether or not the passenger is a non-target passenger, and if it determines that the passenger is a non-target passenger, it may determine that the passenger is not present, and output passenger status information to the effect that the passenger is not present to the driving assistance device 10.

[0075] In this way, even if the passenger's response capability determination state is set as a passenger attribute, the driving assistance device 10 can prevent a situation in which it is meaningless to notify the passenger of a response to the driver's decreased level of alertness when the driver's level of alertness of the vehicle is decreased. As a result, the driving assistance device 10 can prevent the passenger from being unnecessarily bothered. When the driver's level of alertness is decreased, the driving assistance device 10 can notify the passenger of a suggestion if the passenger is not a non-target passenger and is in a state where the passenger is able to take action on the driver, and can not notify the passenger of an unnecessary suggestion if the passenger is a non-target passenger and is in a state where the passenger is unable to take action on the driver.

[0076] <Variation (1-4)> In the first embodiment described above, the notification device 42 is an in-vehicle device installed in the vehicle. However, this is merely an example, and the notification device 42 may be, for example, a device brought into the vehicle cabin by a passenger. For example, the notification device 42 may be an audio device such as headphones held by the passenger, a viewing device, or a mobile information terminal such as a smartphone or tablet. When the passenger is operating or viewing the audio device, viewing device, or mobile information terminal, the assistance control unit 13 may output notification information to the audio device, viewing device, or mobile information terminal, and cause the audio device, viewing device, or mobile information terminal to output audio or display information to prompt the passenger to take measures to prevent deterioration. For example, when the passenger is using the audio device or viewing device in the vehicle cabin, the assistance control unit 13 may interrupt the audio device or viewing device to issue a suggestion notification via audio or display. For example, the assistance control unit 13 detects that the audio device or the viewing device is being used by acquiring information from the audio device or the viewing device indicating that the audio device or the viewing device is being used (hereinafter referred to as "usage information"). When the assistance control unit 13 acquires the usage information from the audio device or the viewing device, it outputs notification information to the audio device or the viewing device. As a result, the assistance control unit 13 controls the audio device or the viewing device to interrupt the audio or the display screen being played back and provide a suggestion notification. Furthermore, for example, when a passenger is performing a visual operation using a mobile information terminal in the vehicle cabin, the assistance control unit 13 may interrupt the mobile information terminal and provide a suggestion notification by audio or display. For example, the assistance control unit 13 detects that the mobile information terminal is being used by acquiring usage information from the mobile information terminal indicating that the mobile information terminal is being used. When the assistance control unit 13 acquires the usage information from the mobile information terminal, it outputs notification information to the mobile information terminal. As a result, the assistance control unit 13 controls the mobile information terminal to interrupt the audio or the display screen being played back and provide a suggestion notification.Furthermore, when the assistance control unit 13 issues the above-described suggestion notification to, for example, an audio device, a viewing device, or a mobile information terminal, it may further output notification information to a vibration generating device serving as the notification device 42 to generate vibrations, making it easier for passengers to notice the suggestion notification.

[0077] In this way, when a passenger is using an audio device, a viewing device, or a mobile information terminal inside the vehicle, the assistance control unit 13 interrupts the audio device, viewing device, or mobile information terminal with a suggestion notification by voice or display, thereby enabling the driving assistance device 10 to more reliably allow the passenger to take measures to prevent a decline in alertness and cooperate in improving the driver's level of alertness.

[0078] <Modification (1-5)> In the above-described first embodiment, in the driving assistance device 10, when the assistance control unit 13 determines that the driver's alertness is less than the first threshold based on the driver state information and when the passenger's alertness based on the passenger state information satisfies the response possibility condition (e.g., <Response possibility condition (1)>), the assistance control unit 13 determines whether to have the passenger take a response to prevent a decline and provides a suggestion notification. The assistance control unit 13 determines whether to have the passenger take a response to prevent a decline based on the passenger's alertness, and when it determines that the passenger should take a response to prevent a decline, the assistance control unit 13 is configured to have the passenger take a predetermined type of response to prevent a decline. Furthermore, the assistance control unit 13 is configured to perform the wakefulness improvement control on the driver in a predetermined type. However, this is merely an example. In the above-described first embodiment, the assistance control unit 13 may determine the mode of the driver's alertness improvement control, whether to implement a countermeasure to prevent a decline in the driver's alertness for the passenger, and the mode of the suggestion notification when the countermeasure is implemented, based on the driver's alertness and the passenger's alertness. For example, the assistance control unit 13 determines the mode of the driver's alertness improvement control, whether to implement a countermeasure to prevent a decline in the driver's alertness for the passenger, and the mode of the suggestion notification when the countermeasure is implemented, based on a mode determination condition. The mode determination condition defines the mode of the driver's alertness improvement control and the mode of the suggestion notification to the passenger, depending on the driver's alertness and the passenger's alertness. The mode determination condition is set, for example, by a developer or the like. After setting the mode determination condition, the developer or the like stores information indicating the mode determination condition (hereinafter referred to as "mode determination condition information") in a buffer or the like internal to the assistance control unit 13. The mode determination conditions may be updateable as needed.

[0079] 7 is a table showing an example of the contents of the mode determination conditions referenced by the driver assistance control unit 13 when the driver assistance control unit 13 determines, based on the driver's and passenger's alertness, the mode of driver alertness improvement control, whether to implement a decline prevention measure for the passenger, and the mode of the decline prevention measure if the decline prevention measure is implemented. For convenience, the driver's and passenger's alertness levels are shown in FIG. 7 as alertness levels. Also, in FIG. 7 , "D:" indicates a stimulus to the driver, i.e., a mode of driver alertness improvement control, and "P:" indicates a notification to the passenger, i.e., a mode of decline prevention measure to be implemented by the passenger through a suggested notification. 7, "vibration" indicates output of alertness enhancement control information or alert information that generates vibration or retracts a seat belt to a vibration generator serving as the alertness enhancement device 41, a seat belt pretensioner serving as the alertness enhancement device 41, a vibration generator serving as the alertness device 42, or a seat belt pretensioner serving as the alertness device 42. Also, in Fig. 7, "audio" indicates output of alertness enhancement control information or alert information that causes a sound or voice to be output to a voice output device serving as the alertness enhancement device 41 or a voice output device serving as the alertness device 42.

[0080] <When the driver's alertness level is weak, DL4> For example, when the driver's alertness level is "DL4," the mode determination conditions associate the mode of the driver's alertness improvement control with whether or not to implement a passenger's alertness reduction prevention measure and the mode of the suggestion notification if the passenger is to implement the measure, as follows, depending on the passenger's alertness level: - When the passenger's alertness level is "DL5" or "DL4," the alertness improvement control is performed to improve the driver's alertness by voice, and the passenger is notified of the suggestion by voice. - When the passenger's alertness level is "DL3," the alertness improvement control is performed to improve the driver's alertness by voice, and the passenger is notified of the suggestion by voice and vibration. - When the passenger's alertness level is "DL2" or "DL1," no suggestion notification is provided to the passenger so as not to bother the passenger, and only the alertness improvement control is performed to improve the driver's alertness by vibration.

[0081] When the driver's level of alertness corresponds to the alertness level "DL4," the driver is estimated to be in a state of slight deactivation. For example, the level of alertness is less than the first threshold and greater than or equal to the third threshold (first threshold > third threshold). Here, the third threshold is, for example, 0.6. When the passenger's level of alertness corresponds to the alertness level "DL2" or "DL1," the passenger is estimated to be in a state of severe deactivation or dozing. For example, the level of alertness is less than or equal to the fourth threshold (first threshold > fourth threshold). Here, the fourth threshold is, for example, 0.5. In this case, the driver's level of deactivation is not so severe that it significantly interferes with driving, and it is expected that the driver's level of alertness can be improved, for example, by simply performing alertness improvement control without the passenger's cooperation. On the other hand, the passenger is estimated to be in a state of severe deactivation or dozing. If the driver's alertness improvement control were to be performed in a manner that could be noticed by passengers, such as by voice, there is a risk of unnecessarily waking up passengers who do not need to be encouraged to improve the driver's alertness. Therefore, the mode determination condition is set so that, for example, when the driver's alertness level is "DL4" and the passenger's alertness level is "DL2" or "DL1," the driver assistance control unit 13 does not provide a suggestion notification to the passengers to avoid disturbing them, and only performs the alertness improvement control to improve the driver's alertness using vibration. In other words, when the driver's alertness level is estimated to be a level that indicates a weakly decreased state of alertness and the passenger's alertness level is estimated to be a level that indicates a strong decreased state of alertness or a state of drowsiness, the assistance control unit 13 determines that the alertness improvement control mode will not disturb the passengers and will not perform a decrease prevention measure. In this way, the driving assistance device 10 can prevent unnecessarily waking up passengers who do not need to be encouraged to improve the driver's alertness.

[0082] <When the driver's alertness level is moderately reduced DL3> For example, when the driver's alertness level is "DL3," the mode determination conditions associate the mode of the driver's alertness improvement control with whether or not to implement measures to prevent a decrease in the level of the passenger and the mode of the suggested notification if the measures to prevent a decrease are implemented, depending on the passenger's alertness level, as follows: Note that when the driver's alertness level is "DL3," improving the driver's alertness is important. - Regardless of the passenger's alertness level, i.e., without considering that the passenger may be woken up by the audio when the passenger's alertness level is low, the alertness improvement control is performed to improve the driver's alertness by audio and vibration. Regarding the notification of suggestions to passengers, if the passenger's alertness level is "DL5" or "DL4", a sound notification will be made; if the passenger's alertness level is "DL3", a sound and vibration notification will be made; and if the passenger's alertness level is "DL2", "DL1" or "DL1", no suggestion notification will be made to the passenger.

[0083] <When the driver's alertness level is DL2, with a strong decrease in alertness> For example, when the driver's alertness level is "DL2," the mode determination condition associates the mode of the driver's alertness improvement control with whether or not to implement measures to prevent a decrease in the alertness of the passenger and the mode of the suggestion notification if the measures to prevent a decrease are implemented, as follows, depending on the passenger's alertness level. Note that when the driver's alertness level is "DL2," improving the driver's alertness is an urgent matter. - Regardless of the passenger's alertness level, alertness improvement control is performed to improve the driver's alertness by providing strong stimuli (e.g., loud audio, strong vibration) to the driver. - Regarding the suggestion notification to the passenger, when the passenger's alertness level is "DL5" or "DL4," a suggestion notification is made using a loud audio sound. When the passenger's alertness level is "DL3" or "DL2," a suggestion notification is made using a loud audio sound and strong vibration. When the passenger's alertness level is "DL1," no suggestion notification is made to the passenger.

[0084] <When the driver falls asleep (DL1, 0)> For example, when the driver's alertness level is "DL1" or "DL0," the mode determination conditions associate the mode of driver alertness improvement control with whether or not to implement measures to prevent a decrease in the driver's alertness and the mode of suggestion notification if the measures to prevent a decrease are implemented, depending on the passenger's alertness level, as follows: When the driver's alertness level is "DL1" or "DL0," the driver can be said to be in a dozing state. The mode of driver alertness improvement control is the same as the mode of alertness improvement control when the driver's alertness level is "DL2." Furthermore, the mode of suggestion notification to the passenger is the same as the mode of suggestion notification when the driver's alertness level is "DL2."

[0085] If the alarm device 42 does not include a vibration generator, the assistance control unit 13 may, for example, replace "vibration" or "strong vibration" in the above-described condition for determining the mode with "alarm sound" or "strong alarm sound," etc. Here, it is assumed that the alarm device 42 includes a sound output device. The alarm sound is, for example, a buzzer sound, a sound effect, or a sound with a changing tone.

[0086] In the configuration of the <Modification (1-5)>, the operation of the driving assistance device 10 is, in the process of determining whether or not a suggestion notification is required for the passenger in step ST40 described in the flowchart of Fig. 3 , the operation shown in the flowchart of Fig. 8 is performed instead of the operation shown in the flowchart of Fig. 4 . Fig. 8 is a flowchart for explaining details of the operation of the process of determining whether or not a suggestion notification is required that the driving assistance device 10 performs in step ST40 of Fig. 3 when the assistance control unit 13 determines, based on the driver's alertness and the passenger's alertness, the mode of driver alertness improvement control, whether or not to perform a decrease prevention measure for the passenger, and the mode of suggestion notification if the decrease prevention measure is performed, based on the driver's alertness and the passenger's alertness in the first embodiment. The details of the processes of steps ST1001 and ST1004 of Fig. 8 are the same as the details of the processes of steps ST1001 and ST1004 of Fig. 4 , respectively, and therefore will not be described again. If it is determined in step ST1001 that a passenger is present (if "YES" in step ST1001), the assistance control unit 13 determines the type of alertness improvement control for the driver, whether to take measures to prevent a decline in the driver's alertness, and the type of suggested notification if such measures are taken, based on the driver's alertness and the passenger's alertness (step ST2001).

[0087] In the operation of the driving assistance device 10 described using the flowchart of Figure 3, for example, if it is determined in step ST30 that the driver's alertness is less than the first threshold, the assistance control unit 13 may output alertness improvement control information to the alertness improvement device 41 before processing of step ST40 is performed. However, in this <Variant Example (1-5)>, in the flowchart of Figure 3, the assistance control unit 13 outputs alertness improvement control information to the alertness improvement device 41 after processing of step ST50.

[0088] In this way, the assistance control unit 13 determines the type of control to increase the driver's level of alertness, whether to take measures to prevent a decline in the level of alertness for the passenger, and the type of suggestion notification to be given if such measures to prevent a decline are to be taken, based on the driver's level of alertness and the passenger's level of alertness, so that the driving assistance device 10 can perform control to increase the driver's level of alertness and provide suggestion notifications to the passengers that are more in line with the state of the driver and passenger.

[0089] <Modification (1-6)> In the above-described first embodiment, the driver state detection unit 211 and the passenger state detection unit 212 in the state detection device 20 are each connected to a separate sensor (the first sensor 31 or the second sensor 32), but this is merely an example. For example, one sensor capable of simultaneously detecting the driver and the passenger may be connected to the state detection device 20, and the state detection unit 21 in the state detection device 20 may detect the states of the driver and the passenger based on sensor information in which the driver and the passenger are detected from the one sensor.

[0090] FIG. 9 is a diagram illustrating an example configuration of a driving assistance system 1 in which one sensor capable of simultaneously detecting the driver and passengers is connected to the state detection device 20. As illustrated in FIG. 9 , in the driving assistance system 1, the state detection device 20 may be connected to one third sensor 33. The third sensor 33 is a sensor capable of simultaneously detecting the driver and passengers. The third sensor 33 may be, for example, an imaging device, a biometric sensor, or a millimeter-wave radar. Here, assuming that the third sensor 33 is an imaging device, the third sensor 33 is provided, for example, in the center of the vehicle in the vehicle width direction so as to be able to capture images of the driver and passengers. Note that, in the first embodiment, the "center" in the vehicle width direction is not limited to the strict "center" but also includes "approximately the center." Specifically, the third sensor 33 is installed in the vehicle's overhead console, center console, or near the center of the dashboard where a car navigation system or the like is installed. Note that the third sensor 33 may be an imaging device capable of simultaneously capturing images of the driver and passengers in the rear seats.

[0091] For example, in the state detection unit 21 of the state detection device 20, the driver state detection unit 211 acquires sensor information (hereinafter referred to as "third sensor information") from the third sensor 33, in this case, an image (hereinafter referred to as "third captured image") of the driver captured by an imaging device serving as the third sensor 33, and detects the state of the driver based on the acquired third captured image. Also, the passenger state detection unit 212 detects the state of the passenger based on the third captured image.

[0092] For example, in the state detection device 20, the state detection unit 21 may detect the states of the driver and the passenger based on the third captured image. In this case, it is not essential that the state detection unit 21 includes the driver state detection unit 211 and the passenger state detection unit 212.

[0093] In this way, in the driving assistance system 1, the state detection device 20 is connected to one third sensor 33 and detects the state of the driver and passengers based on the third sensor information obtained from the third sensor 33, thereby enabling the driving assistance system 1 to reduce the hardware cost of the state detection device 20.

[0094] <Modification (1-7)> In the first embodiment described above, for example, the driving assistance device 10 may be connected to a storage device (not shown), and the assistance control unit 13 in the driving assistance device 10 may store information (hereinafter referred to as "assistance content information") in the storage device, including captured images of the interior and exterior of the vehicle, the driver's level of alertness, or information regarding whether a suggestion notification was executed and the content of the notification. More specifically, the content of the suggestion notification may be, for example, the type of notification information output to the notification device 42. For example, the assistance control unit 13 may acquire, from an exterior image capture device (not shown) installed in the vehicle, captured images of the exterior of the vehicle (hereinafter referred to as "exterior image capture"), and include the acquired images in the assistance content information. Furthermore, the assistance control unit 13 may include the first captured image output from the first sensor 31 and the second captured image output from the second sensor 32 as captured images of the interior of the vehicle (hereinafter referred to as "interior image capture"). If the first sensor 31 or the second sensor 32 is not an imaging device, the assistance control unit 13 may, for example, acquire an in-vehicle image captured by an in-vehicle imaging device (not shown) installed in the vehicle, and include the acquired image in the assistance content information.

[0095] In the driving assistance device 10, the assistance control unit 13 stores assistance content information in a storage device, so that the driving assistance device 10 can provide the user, etc. with information that will help verify the behavior of the occupant when a decrease in the driver's alertness is detected, the behavior of the alertness improvement device 41, the behavior of the alarm device 42, or the effect of improving the driver's alertness, etc.

[0096] <Modification (1-8)> In the above-described <Modification (1-5)>, the assistance control unit 13 may increase the volume of the suggestion notification to the passenger, for example, when the passenger is in a daze or gazing at the scenery outside, or in a so-called inattentive state. To give a specific example, the assistance control unit 13 determines to provide the passenger with a voice suggestion notification because, according to the mode determination conditions, the driver's alertness level is equivalent to the alertness level "DL3" and the passenger's alertness level is equivalent to the alertness level "DL4." In this case, the assistance control unit 13 determines to provide the passenger with a voice suggestion notification at a higher volume when the passenger is in a daze or in an inattentive state. Note that in this case, the passenger's state detected by the passenger state detection unit 212 includes the passenger's behavior. The passenger's behavior includes, for example, a daze or an inattentive state. The passenger state detection unit 212 may detect whether the passenger is in a daze or looking aside by, for example, performing known image processing on the second captured image. Alternatively, the passenger state detection unit 212 may detect whether the passenger is in a daze or looking aside by using a machine learning model. The assistance control unit 13 may determine whether the passenger is in a daze or looking aside from the passenger state information output from the passenger state detection unit 212.

[0097] In this way, if the assistance control unit 13 detects that the passenger is in a daze or looking away, the assistance control unit 13 strengthens the suggestion notification to the passenger, thereby enabling the driving assistance device 10 to more reliably prompt the passenger to take measures to prevent a decline.

[0098] 10A and 10B are diagrams illustrating an example of a hardware configuration of the driving assistance device 10 according to the first embodiment.

[0099] In the first embodiment, the functions of the driver state information acquisition unit 11, the passenger state information acquisition unit 12, and the assistance control unit 13 are realized by the processing circuit 101. That is, the driving assistance device 10 includes the processing circuit 101 for controlling the driver to improve his / her alertness and the passenger to receive a suggestion notification when the alertness of the driver of the vehicle is reduced.

[0100] The processing circuitry 101 may be dedicated hardware as shown in FIG. 10A, or may be a processor 104 that executes a program stored in memory as shown in FIG. 10B.

[0101] When processing circuitry 101 is dedicated hardware, processing circuitry 101 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0102] When the processing circuit is the processor 104, the functions of the driver status information acquisition unit 11, the passenger status information acquisition unit 12, and the assistance control unit 13 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 105. The processor 104 executes the functions of the driver status information acquisition unit 11, the passenger status information acquisition unit 12, and the assistance control unit 13 by reading and executing the program stored in the memory 105. That is, the driving assistance device 10 includes the memory 105 for storing a program that, when executed by the processor 104, results in the execution of steps ST10 to ST70 in FIG. 3 , steps ST10 to ST92 in FIG. 5 , or steps ST10 to ST101 in FIG. 6 . The program stored in the memory 105 can also be said to cause a computer to execute the processing procedures or methods of the driver status information acquisition unit 11, the passenger status information acquisition unit 12, and the assistance control unit 13. Here, the memory 105 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0103] It is also possible to realize some of the functions of the driver state information acquisition unit 11, the passenger state information acquisition unit 12, and the assistance control unit 13 with dedicated hardware and some with software or firmware. For example, the functions of the driver state information acquisition unit 11 and the passenger state information acquisition unit 12 can be realized by the processing circuit 101 as dedicated hardware, and the function of the assistance control unit 13 can be realized by the processor 104 reading and executing a program stored in the memory 105.

[0104] The driving assistance device 10 also includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as a state detection device 20, an alertness enhancement device 41, an alarm device 42, or a storage device (not shown).

[0105] The hardware configuration of the state detection device 20 according to the first embodiment is the same as that shown in FIGS. 10A and 10B , and is therefore not shown. In the first embodiment, the functions of the driver state detection unit 211 and the passenger state detection unit 212 are realized by a processing circuit 101. That is, the state detection device 20 includes the processing circuit 101 for controlling the detection of the states of the driver and passenger. The processing circuit 101 may be dedicated hardware as shown in FIG. 10A , or may be a processor 104 that executes a program stored in a memory as shown in FIG. 10B .

[0106] The processing circuit 101 executes the functions of the driver state detection unit 211 and the passenger state detection unit 212 by reading and executing the programs stored in the memory 105. That is, the state detection device 20 includes the memory 105 for storing programs that, when executed by the processing circuit 101, result in the execution of a process for detecting the driver state or a process for detecting the passenger state. It can also be said that the programs stored in the memory 105 cause a computer to execute the processing procedures or methods of the driver state detection unit 211 and the passenger state detection unit 212. The state detection device 20 includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as the first sensor 31, the second sensor 32, the driving assistance device 10, or the third sensor 33.

[0107] In the first embodiment described above, the driving assistance device 10 is an in-vehicle device, and the driver status information acquisition unit 11, the passenger status information acquisition unit 12, and the assistance control unit 13 are provided in the in-vehicle device. However, the present invention is not limited to this. A system may be configured by the in-vehicle device and the server, with some of the driver status information acquisition unit 11, the passenger status information acquisition unit 12, and the assistance control unit 13 being mounted in the in-vehicle device of the vehicle and the others being provided in a server connected to the in-vehicle device via a network. Alternatively, the driver status information acquisition unit 11, the passenger status information acquisition unit 12, and the assistance control unit 13 may all be provided in the server.

[0108] In the first embodiment, the state detection device 20 is an in-vehicle device, and the driver state detection unit 211 and the passenger state detection unit 212 are provided in the in-vehicle device. However, the present invention is not limited to this. A system may be configured by the in-vehicle device and the server, with some of the driver state detection unit 211 and the passenger state detection unit 212 being provided in the in-vehicle device of the vehicle and the others being provided in a server connected to the in-vehicle device via a network. Alternatively, both the driver state detection unit 211 and the passenger state detection unit 212 may be provided in the server.

[0109] As described above, according to the first embodiment, the driving assistance device 10 is configured to include the driver state information acquisition unit 11 that acquires driver state information related to the state of the driver, including driver alertness information indicating the driver's alertness, the passenger state information acquisition unit 12 that acquires passenger state information related to the state of the passenger, and the assistance control unit 13 that, when it is determined based on the driver state information that the driver's alertness is less than a first threshold, and when it is determined based on the passenger state information that the passenger should take measures to prevent a decrease in the driver's alertness and whether or not to have the passenger take measures to prevent a decrease in the driver's alertness, and when it is determined that the passenger should take measures to prevent a decrease in the driver's alertness, performs a suggestion notification by outputting notification information to the passenger, and when it is determined based on the driver state information that the driver's alertness is less than the first threshold but not to have the passenger take measures to prevent a decrease in the driver's alertness, does not perform a suggestion notification to the passenger. Therefore, the driving assistance device 10 can notify the passenger to take measures to prevent a decrease in the driver's alertness without generating unnecessary notifications.

[0110] More specifically, in the driving assistance device 10, when the assistance control unit 13 determines that the driver's alertness is less than the first threshold based on the driver state information and when the passenger's state based on the passenger state information satisfies the response condition, the assistance control unit 13 determines that the passenger should take measures to prevent a decline in alertness and issues a notification suggesting the action to the passenger, and when the assistance control unit 13 determines that the driver's alertness is less than the first threshold based on the driver state information but the passenger's state does not satisfy the response condition, the assistance control unit 13 determines that the passenger should not take measures to prevent a decline in alertness and does not issue a notification suggesting the action to the passenger. Therefore, the driving assistance device 10 can notify the passenger to take measures to prevent a decline in the driver's alertness without issuing an unnecessary notification.

[0111] Embodiment 2 In the first embodiment, even if the driver's alertness level has decreased, the driving assistance device does not notify the passenger of a suggestion if there is no passenger present, or if there is a passenger present but the passenger is a non-target passenger. In other words, when the driver's alertness level has decreased, no encouragement is given to anyone other than the passenger to improve the driver's alertness. In the second embodiment, an embodiment will be described in which encouragement is given to someone other than the passenger to improve the driver's alertness.

[0112] In the following embodiment 2, as in embodiment 1, the first sensor 31 and the second sensor 32 are assumed to be imaging devices, as an example. Also, in the following embodiment 2, as in embodiment 1, as an example, the response availability determination state is assumed to be the level of wakefulness.

[0113] FIG. 11 is a diagram illustrating a configuration example of a driving assistance device 10a according to a second embodiment. The driving assistance device 10a and the state detection device 20 constitute a driving assistance system 1a. In the second embodiment, the driving assistance device 10a is connected to a conversation device 50 in addition to the state detection device 20, the alertness enhancement device 41, and the notification device 42. The conversation device 50 is mounted, for example, on a vehicle. The conversation device 50 is a device capable of conversing with a driver using AI or the like. Devices capable of conversing with people using AI or the like are well-known devices, and detailed description thereof will be omitted. However, the conversation device 50 may, for example, hold conversations on various topics or topics of interest to the driver. Note that the conversation AI mounted on the conversation device 50 may be implemented as a standalone device on an in-vehicle device, or may be configured to hold conversations using an AI conversation engine (program or application) on a service server. Various topics are updated daily on the server side, so the conversation AI can provide topics of interest to the driver. Furthermore, the conversation AI can pick out topics that match the driver's preferences.

[0114] 11, the same reference numerals are assigned to the same configuration examples as those of the driving assistance device 10 described in the first embodiment using FIG. 1, and duplicated explanations will be omitted. In the driving assistance device 10a according to the second embodiment, the specific operation of the assistance control unit 13a is different from the specific operation of the assistance control unit 13 in the driving assistance device 10 according to the first embodiment.

[0115] The assistance control unit 13a has the following functions in addition to the functions of the assistance control unit 13 already described in the first embodiment. Other functions of the assistance control unit 13a are similar to those of the assistance control unit 13 already described in the first embodiment, and therefore redundant description will be omitted. If there is no passenger present, or if there is a passenger present but the passenger's alertness does not satisfy the above-mentioned <Response Availability Condition (1)>, the assistance control unit 13a does not cause the passenger to take measures to prevent a decrease in alertness; in other words, it determines that the vehicle is not in a state where it can take action to improve the driver's alertness, and does not notify the passenger of a suggestion. In this case, the assistance control unit 13a outputs information (hereinafter referred to as "conversation execution control information") for controlling the conversation device 50 to execute a conversation with the driver. When the conversation execution control information is output from the assistance control unit 13a, the conversation device 50 executes a conversation with the driver.

[0116] The operation of the driving assistance device 10a according to the second embodiment will now be described. Fig. 12 is a flowchart for explaining the operation of the driving assistance device 10a according to the first embodiment. Regarding the operation of the driving assistance device 10a shown in the flowchart of Fig. 12, the details of the processing of steps ST10 to ST80 are the same as the details of the processing of steps ST10 to ST80 already explained using the flowchart of Fig. 3 in the first embodiment, and therefore, redundant explanations will be omitted.

[0117] After outputting the awakening level improvement control information for performing the awakening level improvement control for the driver in step ST80, the assistance control unit 13a outputs conversation execution control information to the conversation device 50 (step ST110).

[0118] In this way, when the driving assistance device 10a determines that the driver's alertness is less than the first threshold based on the driver state information and the passenger's state based on the passenger state information (here, the passenger's alertness) satisfies the response possibility condition, the driving assistance device 10a determines that the passenger should take a response to prevent a decline and issues a suggestion notification. On the other hand, when the driving assistance device 10a determines that the driver's alertness is less than the first threshold based on the driver state information but the passenger's alertness does not satisfy the response possibility condition, the driving assistance device 10a determines that the passenger should not take a response to prevent a decline and does not issue a suggestion notification to the passenger. When the driving assistance device 10a determines that the driver's alertness is less than the first threshold based on the driver state information and, as a result of determining whether to have the passenger take a response to prevent a decline based on the passenger state information, determines that the passenger should not take a response to prevent a decline, the driving assistance device 10a outputs conversation execution control information for controlling the conversation device 50 to execute a conversation with the driver.

[0119] This allows the driving assistance device 10a to prevent a situation in which it is pointless to notify the passenger of a decrease in the driver's alertness when the driver's alertness is low. As a result, the driving assistance device 10a can prevent the passenger from being unnecessarily bothered. When the driver's alertness is low, the driving assistance device 10a notifies the passenger of a suggestion if the passenger's alertness is equal to or greater than the second threshold and the passenger is able to intervene with the driver, and does not notify the passenger of an unnecessary suggestion if the passenger's alertness is below the second threshold and the passenger is unable to intervene with the driver. Furthermore, the driving assistance device 10a can improve the driver's alertness through conversation using the conversation device 50 even when there is no passenger or when the passenger is unable to intervene with the driver. As described above, for example, the conversation AI installed in the conversation device 50 can provide topics of interest to the driver. Furthermore, the conversation AI can pick out topics that match the driver's preferences. Therefore, the driving assistance device 10 can be expected to have an awakening effect on the driver by having the conversation device 50 talk to the driver.

[0120] <Modification (2-1)> In the second embodiment described above, the driving assistance device 10a has the mechanical conversation device 50 conduct a conversation with the driver when there is no passenger or when the passenger is unable to take action against the driver. However, this is merely an example, and the driving assistance device 10a may have a registered person who has been registered in advance as a person who will take action to prevent a decline in driver awareness conduct a conversation with the driver. The registered person may be, for example, a person who the driver has registered in advance as a predetermined conversation partner, or a person in charge of a conversation service provider. Specifically, in the driving assistance device 10a, when there is no passenger or when the passenger's alertness level does not satisfy the above <Response Availability Condition (1)>, the assistance control unit 13a does not have the passenger conduct a response to prevent a decline in driver awareness. In other words, the assistance control unit 13a determines that the driver is not in a state where it is possible to take action to improve the driver's alertness, and does not provide a suggestion notification to the passenger but instead notifies the registered person that the driver's alertness level is declining. More specifically, when the assistance control unit 13a determines that the passenger is not in a state where he or she can take action to improve the driver's level of alertness, the assistance control unit 13a sends a notification to a mobile information terminal, such as a smartphone or tablet terminal, held by the registrant, informing the registrant that the driver's level of alertness is declining. The registrant who receives the notification then communicates with the driver, for example. Alternatively, for example, the assistance control unit 13a may first send a notification to the registrant, and if the registrant and the driver cannot communicate with each other, output conversation execution control information to the conversation device 50. An example of a situation in which the registrant and the driver cannot communicate with each other is when the driving assistance device 10a is unable to notify the registrant due to a communication error between the driving assistance device 10a and the mobile information terminal held by the registrant.

[0121] In this way, when the assistance control unit 13a determines that the driver's alertness is less than the first threshold based on the driver state information, and determines based on the passenger state information whether to have the passenger take measures to prevent a decline in alertness, and determines not to have the passenger take measures to prevent a decline in alertness, the driving assistance control unit 13a may send a notification to a registered person who is registered as a person who will take measures to prevent a decline in alertness, informing them that the driver's alertness is declining. In this way, the driving assistance device 10a encourages the driver to have a conversation with a person that is not mechanical, which is expected to have a more waking effect on the driver than a mechanical conversation.

[0122] Note that the above-described <Modification (1-1)> to <Modification (1-8)> of Embodiment 1 may be applied to the above-described Embodiment 2. For example, when <Modification (1-7)> of Embodiment 1 is applied to Embodiment 2, the assistance control unit 13a may include the conversation execution control information output to the conversation device 50 in the assistance content information to be stored in the storage device.

[0123] The hardware configuration of the driving assistance device 10a according to the second embodiment is the same as that of the driving assistance device 10 according to the first embodiment, as shown in FIGS. 10A and 10B , and is therefore not shown. In the second embodiment, the functions of the driver state information acquisition unit 11, the passenger state information acquisition unit 12, and the assistance control unit 13a are realized by a processing circuit 101. That is, the driving assistance device 10a includes a processing circuit 101 for, when the driver's alertness level of the vehicle driver is low, performing an alertness improvement control for the driver, making a suggestion notification to the passenger, and, when the suggestion notification to the passenger is not made, controlling the conversation device 50 to have a conversation with the driver. The processing circuit 101 may be dedicated hardware as shown in FIG. 10A , or a processor 104 that executes a program stored in a memory as shown in FIG. 10B .

[0124] The processing circuit 101 executes the functions of the driver state information acquisition unit 11, the fellow passenger state information acquisition unit 12, and the assistance control unit 13a by reading and executing a program stored in the memory 105. That is, the driving assistance device 10a includes the memory 105 for storing a program that, when executed by the processing circuit 101, results in the execution of the processes of steps ST10 to ST110 of FIG. 12 described above. It can also be said that the program stored in the memory 105 causes a computer to execute the procedures or methods of the processes of the driver state information acquisition unit 11, the fellow passenger state information acquisition unit 12, and the assistance control unit 13a. The driving assistance device 10a includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as the state detection device 20, the alertness enhancement device 41, the alarm device 42, a storage device (not shown), or the conversation device 50.

[0125] As described above, according to the second embodiment, in the driving assistance device 10a, when it is determined that the driver's alertness is less than the first threshold based on the driver state information, and when it is determined not to cause the passenger to take a decrease prevention measure as a result of determining whether to cause the passenger to take a decrease prevention measure based on the passenger state information, the assistance control unit 13a is configured to output conversation execution control information that controls the conversation device 50 to execute a conversation with the driver. Therefore, even when the passenger is not present or when the passenger is in a state where they are unable to intervene with the driver, the driving assistance device 10a can improve the driver's alertness by conversation via the conversation device 50.

[0126] Embodiment 3 In Embodiment 1, the driving assistance device does not perform automatic driving control of the vehicle. In Embodiment 3, an embodiment will be described in which the driving assistance device performs automatic driving control of the vehicle.

[0127] In the following embodiment 3, as in embodiment 1, the first sensor 31 and the second sensor 32 are assumed to be imaging devices, as an example. Also, in the following embodiment 3, as in embodiment 1, the response feasibility determination state is assumed to be a wakefulness level, as an example. Note that in the following embodiment 3, it is assumed that the vehicle is an autonomously driven vehicle capable of autonomous driving.

[0128] FIG. 13 is a diagram illustrating a configuration example of a driving assistance device 10b according to a third embodiment. The driving assistance device 10b and the state detection device 20 constitute a driving assistance system 1b. In the third embodiment, the driving assistance device 10b is connected to an automatic driving control device 60 in addition to the state detection device 20, the alertness enhancement device 41, and the alarm device 42. The automatic driving control device 60 is mounted, for example, on a vehicle. The automatic driving control device 60 performs automatic driving control of the vehicle, for example, by implementing automatic driving levels 1 to 5. Automatic driving level 1 automates part of the driving, automatic driving level 2 automates multiple driving functions simultaneously, automatic driving level 3 automates driving completely under specific conditions, and automatic driving level 4 is a level capable of fully automatic driving in most situations. It is assumed that automatic driving level 5 is capable of fully automatic driving in all situations. Note that, while the automatic driving control device 60 is mounted on a vehicle here, this is merely an example, and the automatic driving control device 60 may be mounted, for example, on a server.

[0129] 13, the same reference numerals are assigned to the same configuration examples as those of the driving assistance device 10 described in the first embodiment using FIG. 1, and duplicated explanations will be omitted. In the driving assistance device 10b according to the third embodiment, the specific operation of the assistance control unit 13b is different from the specific operation of the assistance control unit 13 in the driving assistance device 10 according to the first embodiment.

[0130] The assist control unit 13b has the following functions in addition to the functions of the assist control unit 13 already described in embodiment 1. Other functions of the assist control unit 13b are similar to the functions of the assist control unit 13 already described in embodiment 1, and therefore redundant description will be omitted. The assist control unit 13b outputs information (hereinafter referred to as "automatic driving control information") to the automatic driving control device 60 to execute automatic driving control of the vehicle based on the driver's alertness. The assist control unit 13b determines the content of automatic driving control in accordance with the automatic driving control determination conditions, and outputs automatic driving control information for causing the automatic driving control device 60 to execute the determined automatic driving control. The automatic driving control determination conditions are conditions that define what type of automatic driving control is to be performed depending on the driver's alertness, and are set in advance by developers, etc. Once the developers, etc. set the automatic driving control determination conditions, they store information indicating the automatic driving control determination conditions (hereinafter referred to as "automatic driving control determination condition information") in an internal buffer, etc., of the assist control unit 13b. The assistance control unit 13b determines the content of the autonomous driving control by referring to the autonomous driving control determination condition information. The autonomous driving control determination conditions include, for example, the following conditions: Note that, for convenience, the driver's alertness will be indicated by an alertness level below.

[0131] - If the alertness level is "DL5" or "DL4", autonomous driving control is not executed. - If the alertness level is "DL3", ACC (Adaptive Cruise Control), AEB (Autonomous Emergency Braking), or LKAS (Lane Keeping Assist System) is activated. - If the alertness level is "DL2", a warning is issued to the driver and autonomous driving control of autonomous driving level 3 or higher is executed. - If the alertness level is "DL1" or "DL0", fully autonomous driving control is executed and the vehicle is parked and stopped in an appropriate location.

[0132] For example, if the driver's alertness level is "DL2," the assistance control unit 13b issues a warning to the driver and outputs, to the automatic driving control device 60, automatic driving control information for executing automatic driving control of automatic driving level 3 or higher. The automatic driving control device 60 outputs a warning to the driver and executes automatic driving control of automatic driving level 3 or higher in accordance with the automatic driving control information. Note that the automatic driving control decision conditions described above are merely examples, and other conditions may be set as the automatic driving control decision conditions. For example, automatic driving control decision conditions may be set to implement various driving assistance depending on the situation around the vehicle. As a specific example, automatic driving control decision conditions may be set to execute automatic driving control such as controlling the vehicle to prevent the vehicle from coming close to a vehicle traveling in an adjacent lane, or maintaining a safe distance when approaching a merging point, depending on the driver's alertness.

[0133] Furthermore, the assistance control unit 13b performs wakefulness improvement control in accordance with the wakefulness of the driver. That is, for example, according to the above-described automatic driving control determination condition, when the wakefulness level of the driver is "DL4," the assistance control unit 13b does not execute the automatic driving control, but performs the wakefulness improvement control to improve the wakefulness of the driver.

[0134] The operation of the driving assistance device 10b according to embodiment 3 will now be described. Fig. 14 is a flowchart for explaining the operation of the driving assistance device 10b according to embodiment 3. Regarding the operation of the driving assistance device 10b shown in the flowchart of Fig. 14, the details of the processing of steps ST10 to ST80 are the same as the details of the processing of steps ST10 to ST80 already explained using the flowchart of Fig. 3 in embodiment 1, and therefore redundant explanations will be omitted.

[0135] When the assistance control unit 13b outputs notification information to the passenger in step ST70 to take measures to prevent a decline, or when it outputs alertness improvement control information to the driver in step ST80 to perform alertness improvement control, it outputs automatic driving control information to the automatic driving control device 60 depending on the driver's alertness (step ST120).

[0136] 14, the operation of the driving assistance device 10b is performed after the processing of step ST70 and step ST80, but this is merely an example. The processing of step ST120 may be performed after the processing of step ST60 and after the processing of step ST80, for example.

[0137] In this way, when the driving assistance device 10b determines that the driver's alertness is less than the first threshold based on the driver state information and the passenger's state based on the passenger state information (here, the passenger's alertness) satisfies the response condition, the driving assistance device 10b determines that the passenger should take measures to prevent a decline and issues a notification suggesting such measures.On the other hand, when the driving assistance device 10b determines that the driver's alertness is less than the first threshold based on the driver state information but the passenger's alertness does not satisfy the response condition, the driving assistance device 10b determines that the passenger should not take measures to prevent a decline and does not issue a notification suggesting such measures to the passenger.Furthermore, the driving assistance device 10b's assistance control unit 13b outputs automatic driving control information to the automatic driving control device 60 based on the driver's alertness.

[0138] This allows the driving assistance device 10b to prevent a situation in which it is meaningless to provide a notification to the passenger to address the driver's decreased level of alertness when the driver's level of alertness is decreased. As a result, the driving assistance device 10b can prevent the passenger from being unnecessarily bothered. When the driver's level of alertness is decreased, the driving assistance device 10b can provide a suggestion notification to the passenger if the passenger's level of alertness is equal to or greater than the second threshold and the passenger is able to intervene with the driver, and can avoid providing an unnecessary suggestion notification to the passenger if the passenger's level of alertness is less than the second threshold and the passenger is unable to intervene with the driver. Furthermore, by being able to output automatic driving control information, the driving assistance device 10b can provide driving assistance that is more prepared for unforeseen circumstances.

[0139] <Variation (3-1)> In the third embodiment described above, when the driver's alertness level is a level at which the driver is estimated to be dozing (e.g., alertness level "DL1" or "DL0") and the assistance control unit 13b determines to perform fully automated driving control to park and stop the vehicle at an appropriate location, and further when the passenger's alertness level is a level at which the passenger is estimated to be dozing (e.g., alertness level "DL1" or "DL0"), the assistance control unit 13b may output information (hereinafter referred to as "notification information") to the alarm device 42 notifying the passenger that the driver's alertness is decreasing. When the notification information is output, the alarm device 42 may, for example, output an alarm sound or output a voice message urging the passenger to wake up. The voice output from the alarm device 42 allows the passenger to confirm the driver's condition and take appropriate action. For example, if the driver is in a mental or physical abnormality state, the passenger can confirm this and take necessary measures.

[0140] Note that the above-described <Modification (1-1)> to <Modification (1-8)> of the first embodiment may be applied to the third embodiment. For example, when <Modification (1-5)> of the first embodiment is applied to the third embodiment, a condition may be set in the mode determination condition that, if the driver's alertness level is "DL2," the vehicle is automatically stopped regardless of the passenger's alertness level. Furthermore, for example, when <Modification (1-7)> of the first embodiment is applied to the third embodiment, the assistance control unit 13b may include the automatic driving control information output to the automatic driving control device 60 in the assistance content information to be stored in the storage device. Furthermore, the third embodiment may be applied to the second embodiment.

[0141] The hardware configuration of the driving assistance device 10b according to the third embodiment is the same as that of the driving assistance device 10 according to the first embodiment, as shown in FIGS. 10A and 10B , and therefore is not shown in the drawings. In the third embodiment, the functions of the driver state information acquisition unit 11, the passenger state information acquisition unit 12, and the assistance control unit 13b are realized by a processing circuit 101. That is, the driving assistance device 10b includes a processing circuit 101 for, when the driver's alertness level of the vehicle driver is low, performing alertness improvement control for the driver, making a suggestion notification to the passenger, and, when the suggestion notification to the passenger is not made, controlling the conversation device 50 to have a conversation with the driver. The processing circuit 101 includes the processing circuit 101. The processing circuit 101 may be dedicated hardware as shown in FIG. 10A , or may be a processor 104 that executes a program stored in a memory as shown in FIG. 10B .

[0142] The processing circuit 101 reads and executes a program stored in the memory 105 to perform the functions of the driver state information acquisition unit 11, the fellow passenger state information acquisition unit 12, and the assistance control unit 13b. That is, the driving assistance device 10b includes a memory 105 for storing a program that, when executed by the processing circuit 101, results in the execution of the processes of steps ST10 to ST120 in FIG. 14 described above. It can also be said that the program stored in the memory 105 causes a computer to execute the procedures or methods of the processes of the driver state information acquisition unit 11, the fellow passenger state information acquisition unit 12, and the assistance control unit 13b. The driving assistance device 10b includes an input interface device 102 and an output interface device 103 that perform wired or wireless communication with devices such as the state detection device 20, the alertness enhancement device 41, the alarm device 42, a storage device (not shown), or the automatic driving control device 60.

[0143] As described above, according to the third embodiment, the driving assistance device 10b is configured such that the assistance control unit 13b outputs automatic driving control information for performing automatic driving control of the vehicle based on the driver's alertness. Therefore, the driving assistance device 10b can provide driving assistance that is more prepared for unexpected situations.

[0144] It should be noted that the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0145] The driving assistance device of the present disclosure can notify passengers to respond to a decrease in the driver's alertness without providing unnecessary alerts, and can therefore be applied to driving assistance devices that encourage passengers to cooperate in improving the driver's alertness when the driver's alertness decreases.

[0146] 1, 1a, 1b Driving assistance system, 10, 10a, 10b Driving assistance device, 11 Driver state information acquisition unit, 12 Passenger state information acquisition unit, 13, 13a, 13b Assistance control unit, 20 State detection device, 21 State detection unit, 211 Driver state detection unit, 212 Passenger state detection unit, 31 First sensor, 32 Second sensor, 33 Third sensor, 41 Alertness improvement device, 42 Notification device, 50 Conversation device, 60 Automatic driving control device, 101 Processing circuit, 102 Input interface device, 103 Output interface device, 104 Processor, 105 Memory.

Claims

1. A driving assistance device comprising: a driver state information acquisition unit that acquires driver state information regarding the state of the driver including the driver's level of alertness; a passenger state information acquisition unit that acquires passenger state information regarding the state of a passenger; and an assistance control unit that, when it is determined based on the driver state information that the driver's level of alertness is less than a first threshold, and when it is determined based on the passenger state information whether or not to have the passenger take measures to prevent a decline in the driver's level of alertness and it is determined that the passenger should take the measures to prevent a decline, issues a suggestion notification that outputs notification information to the passenger to cause the passenger to take the measures to prevent a decline, and when it is determined based on the driver state information that the driver's level of alertness is less than the first threshold but it is determined that the passenger should not take the measures to prevent a decline, does not issue the suggestion notification to the passenger.

2. The driving assistance device according to claim 1, characterized in that, when the assistance control unit determines that the driver's level of alertness is less than the first threshold based on the driver status information and the passenger's status based on the passenger status information satisfies the response possible condition, the assistance control unit determines that the passenger should take the response to prevent a decline and issues the suggestion notification, and when the passenger's status does not satisfy the response possible condition even when the assistance control unit determines that the driver's level of alertness is less than the first threshold based on the driver status information, the assistance control unit determines that the passenger should not take the response to prevent a decline and does not issue the suggestion notification to the passenger.

3. The driving assistance device according to claim 2, characterized in that the passenger's state includes the passenger's level of alertness, and the assistance control unit determines that the passenger's state satisfies the response possible condition when it determines, based on the passenger state information, that the passenger's level of alertness is equal to or higher than a second threshold.

4. The driving assistance device according to claim 1, characterized in that the response to prevent a decline is to talk to the driver to improve the driver's level of alertness, and the suggestion notification is to output notification information that encourages conversation with the driver.

5. The driving assistance device of claim 1, characterized in that the passenger's state includes the passenger's behavior, and the assistance control unit, after making the suggestion notification to the passenger, makes the suggestion notification to the passenger again if it determines, based on the passenger state information, that the passenger has not taken the measures to prevent the passenger from falling.

6. The driving assistance device according to claim 1, characterized in that, when the suggestion notification is made to the passenger, the assistance control unit detects a change in the driver's alertness before and after making the suggestion notification to the passenger based on the driver state information, and outputs assistance result information regarding the detected change in the driver's alertness.

7. The driving assistance device according to claim 6, characterized in that the assistance control unit detects a change in the driver's alertness before and after the suggestion notification is made to the passenger based on the driver state information, and if the driver's alertness has not improved, outputs the assistance result information informing the passenger that the driver's alertness has not improved.

8. The driving assistance device of claim 2, characterized in that the passenger's state includes attributes of the passenger, and the assistance control unit determines, based on the passenger state information, that the passenger's state does not satisfy the response ability condition if the passenger is a non-target passenger who is assumed to be incapable of responding to the driver's decreased level of alertness.

9. The driving assistance device according to claim 8, wherein the non-target passengers include babies, small children, or people with dementia.

10. The driving assistance device of claim 1, characterized in that, when the passenger is operating or viewing an audio device, a viewing device, or a mobile information terminal, the assistance control unit outputs the notification information to the audio device, the viewing device, or the mobile information terminal, causing an audio output or display to be performed to prompt the passenger to take measures to prevent the deterioration.

11. The driving assistance device of claim 1, characterized in that the state of the passenger includes the passenger's level of alertness, and when the assistance control unit determines that the driver's level of alertness is less than the first threshold, it outputs to the driver level of alertness improvement control information for performing alertness improvement control to improve the driver's level of alertness, and determines, based on the driver's level of alertness and the passenger's level of alertness, the manner of the alertness improvement control for the driver, whether or not to implement the deterioration prevention measure for the passenger, and the manner of the suggested notification if the deterioration prevention measure is to be implemented.

12. The driving assistance device of claim 11, characterized in that, when the driver's level of alertness is such that the driver is estimated to be in a state of slightly decreased alertness and the passenger's level of alertness is such that the passenger is estimated to be in a state of strongly decreased alertness or dozing, the assistance control unit determines that the alertness improvement control will be performed in a manner that does not cause inconvenience to the passenger and will not execute the decrease prevention response.

13. The driving assistance device according to claim 1, characterized in that the assistance control unit outputs automatic driving control information for performing automatic driving control of the vehicle based on the driver's level of alertness.

14. The driving assistance device according to claim 13, characterized in that the assistance control unit outputs the automatic driving control information for performing fully automatic driving control to park and stop the vehicle when the driver's level of alertness is such that the driver is estimated to be dozing off, and when the automatic driving control information for performing the fully automatic driving control is output and the passenger's level of alertness is such that the passenger is estimated to be dozing off, it outputs notification information to notify the passenger that the driver's level of alertness has decreased.

15. The driving assistance device described in claim 1, characterized in that when the assistance control unit determines based on the driver status information that the driver's level of alertness is less than the first threshold, and when the assistance control unit determines based on the passenger status information whether or not to have the passenger take the measure to prevent a decline, and determines not to have the passenger take the measure to prevent a decline, the assistance control unit outputs conversation execution control information that controls a conversation device to carry out a conversation with the driver.

16. The driving assistance device of claim 1, characterized in that when the assistance control unit determines that the driver's level of alertness is less than the first threshold based on the driver status information, and when the assistance control unit determines based on the passenger status information whether or not to have the passenger take the measure to prevent a decline, and determines not to have the passenger take the measure to prevent a decline, it issues a notification to a registered person who is registered as a person who will take the measure to prevent a decline, informing them that the driver's level of alertness has declined.

17. The driving assistance device according to claim 13, characterized in that the assistance control unit records assistance content information including the driver's level of alertness, information regarding whether the suggestion notification is executed and its content, or the automatic driving control information.

18. A driving assist device comprising: a step in which a driver state information acquisition unit acquires driver state information regarding the state of the driver including the driver's level of alertness; a step in which a passenger state information acquisition unit acquires passenger state information regarding the state of the passenger; and a step in which an assist control unit, when it determines based on the driver state information that the driver's level of alertness is less than a first threshold, and when it determines based on the passenger state information whether or not to have the passenger take measures to prevent a decline in the driver's level of alertness and determines to have the passenger take the measures to prevent a decline, issues a suggestion alert by outputting alert information to the passenger to make the measures to prevent a decline, and when it determines based on the driver state information that the driver's level of alertness is less than the first threshold but determines not to have the passenger take the measures to prevent a decline, does not issue the suggestion alert to the passenger.

19. A driving assistance system comprising: a driving assistance device according to any one of claims 1 to 17; and a state detection device having a state detection unit that detects the state of the driver and the state of the passenger based on sensor information.

20. The driving assistance system according to claim 19, characterized in that the state detection unit acquires, as the sensor information, images of the driver and the passenger taken by an imaging device capable of photographing the driver and the passenger, and detects the state of the driver and the state of the passenger based on the captured images.

Images