Notification device, notification method, and notification computer program

Abstract

To appropriately notify a driver of a vehicle of a message so that automatic driving can be continued.SOLUTION: A notification device detects a predetermined action of a driver from a driver image including a face region in which the face of the driver of the vehicle of which the travel is controlled by an automatic driving system is represented, outputs a certainty factor showing certainty in detection, specifies a message for suppressing a control termination action applicable to a termination condition of control by the automatic driving system with possibility of being performed following the detected predetermined action, and notifies the driver of the specified message via the output device when an efficacy level to be calculated by multiplying the certainty factor by first probability by which the control termination action is suppressed when the driver is notified of the specified message or multiplying the certainty factor by second probability by which the control termination action is performed when the driver is not notified of the specified message is larger than a predetermined notification threshold.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a notification device, a notification method, and a notification computer program for notifying a driver of a vehicle of a message. [Background technology]

[0002] Certain behaviors by a vehicle driver, such as dozing off, looking away, and unstable posture, are undesirable for the continued safe driving of the vehicle. The vehicle alarm device described in Patent Document 1 detects when the driver is dozing off or looking away, and issues an alarm to the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-133402 Summary of the Invention [Problem to be solved by the invention]

[0004] In vehicles where the driving is controlled by an automated driving system at an automated driving level where the driver is not required to monitor the surroundings, the driver is permitted to engage in various non-driving behaviors (for example, operating a smartphone). However, non-driving behaviors that make it impossible to confirm that the driver can take over driving from the automated driving system (for example, operating a smartphone so that the driver's eyes are blocked from the driver monitor camera) can lead to a situation where automated driving cannot continue appropriately, and are therefore subject to a warning to end control by the automated driving system. It is not easy for the driver to know in advance what non-driving behaviors will lead to such a situation and be subject to a warning.

[0005] The present disclosure aims to provide a notification device that can appropriately notify a driver of a vehicle of a message so that the driver can continue autonomous driving. [Means for solving the problem]

[0006] The notification device of the present disclosure includes a detection unit that detects a predetermined driver behavior from a driver image including a facial area showing the face of the driver of a vehicle whose driving is controlled by an autonomous driving system, and outputs a certainty factor indicating the likelihood of the detection; an identification unit that identifies a message to suppress a control termination behavior that may be performed following the detected predetermined behavior and that corresponds to a termination condition for control by the autonomous driving system; and a notification unit that notifies the driver of the identified message via an output device when an effectiveness level calculated by multiplying the certainty factor by a first probability that the control termination behavior will be suppressed if the identified message is notified to the driver, or by a second probability that the control termination behavior will be performed if the identified message is not notified to the driver, is greater than a predetermined notification threshold.

[0007] In the notification device according to the present disclosure, it is preferable that the notification unit suppresses notification of the identified message if the elapsed time since the most recent notification of a message identical to the identified message is shorter than a predetermined message interval threshold.

[0008] In the notification device of the present disclosure, it is preferable that the detection unit further detects the driver's control termination action from the driver image, and that the notification unit, in response to the detection of the control termination action, notifies the driver of the termination of control by the autonomous driving system via the output device in a manner that is stronger than a message notification.

[0009] In the notification device according to the present disclosure, it is preferable that the notification unit suppresses notification of the identified message if the elapsed time from the time when the most recent termination notice based on the detection of a control termination action corresponding to the identified message was notified is shorter than a predetermined termination notice interval threshold that is longer than the message interval threshold.

[0010] The notification method disclosed herein includes detecting a predetermined driver behavior from a driver image including a facial area showing the face of the driver of a vehicle whose driving is controlled by an autonomous driving system, outputting a confidence level indicating the likelihood of the detection, identifying a message for suppressing a control termination behavior that may be performed following the detected predetermined behavior and that corresponds to a termination condition for control by the autonomous driving system, and notifying the driver of the identified message via an output device if the effectiveness level calculated by multiplying the confidence level by a first probability that the control termination behavior will be suppressed if the identified message is notified to the driver, or by a second probability that the control termination behavior will be performed if the identified message is not notified to the driver, is greater than a predetermined notification threshold.

[0011] The notification computer program disclosed herein causes a computer to perform the following steps: detect a predetermined behavior of a driver from an image including a facial area showing the face of the driver of a vehicle whose driving is controlled by an autonomous driving system; output a confidence level indicating the likelihood of the detection; identify a message to suppress a control termination behavior that may be performed following the detected predetermined behavior and that corresponds to a termination condition for control by the autonomous driving system; and notify the driver of the identified message via an output device if the effectiveness level calculated by multiplying the confidence level by a first probability that the control termination behavior will be suppressed if the identified message is notified to the driver, or by a second probability that the control termination behavior will be performed if the identified message is not notified to the driver, is greater than a predetermined notification threshold.

[0012] According to the notification device of the present disclosure, it is possible to appropriately notify the driver of a vehicle of a message so that the driver can continue autonomous driving. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic configuration diagram of a vehicle in which a notification device is installed. [Figure 2] FIG. 2 is a hardware schematic diagram of the notification device. [Figure 3]FIG. 10 is a diagram illustrating an overview of a behavior detection table. [Figure 4] FIG. 10 is a diagram illustrating an outline of a control termination action table. [Figure 5] FIG. 10 is a diagram illustrating an overview of a message table. [Figure 6] FIG. 2 is a functional block diagram of a processor included in the notification device. [Figure 7] 10 is a flowchart of a notification process. DETAILED DESCRIPTION OF THE INVENTION

[0014] A notification device capable of appropriately notifying a driver of a vehicle of a message so that the driver can continue autonomous driving will be described in detail below with reference to the drawings. The notification device detects a predetermined driver behavior from a driver image including a facial region representing the face of the driver of a vehicle whose driving is controlled by an autonomous driving system, and outputs a confidence level indicating the likelihood of the detection. The notification device identifies a message for suppressing a control-terminating behavior that may occur following the detected predetermined behavior and that corresponds to a condition for terminating control by the autonomous driving system. The notification device calculates an effectiveness level indicating the effectiveness of the message notification for suppressing the control-terminating behavior following the detected behavior by multiplying the confidence level by a first probability that the control-terminating behavior will be suppressed if the identified message is notified to the driver, or by a second probability that the control-terminating behavior will be performed if the identified message is not notified to the driver. Then, if the calculated effectiveness level is greater than a predetermined notification threshold, the notification device notifies the driver of the identified message via an output device.

[0015] FIG. 1 is a schematic diagram of a vehicle in which a notification device is installed.

[0016] The vehicle 1 has a peripheral camera 2, a driver monitor camera 3, a speaker device 4, a GNSS (Global Navigation Satellite System) receiver 5, a storage device 6, a driving control device 7, and a notification device 8. The peripheral camera 2, the driver monitor camera 3, the speaker device 4, the GNSS (Global Navigation Satellite System) receiver 5, and the storage device 6 are communicably connected to the driving control device 7 and the notification device 8 via an in-vehicle network that complies with a standard such as a controller area network.

[0017] The surrounding camera 2 is an example of a surrounding sensor for generating surrounding data that represents the situation around the vehicle 1. The surrounding camera 2 has a two-dimensional detector configured with an array of photoelectric conversion elements, such as a CCD or C-MOS, that are sensitive to visible light, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The surrounding camera 2 is disposed, for example, at the front upper part of the vehicle interior, facing forward, photographs the situation around the vehicle 1 through the windshield at predetermined photographing intervals (for example, 1 / 30 to 1 / 10 seconds), and outputs the surrounding images as surrounding data that represent the situation around the vehicle 1.

[0018] The driver monitor camera 3 is an example of a driver sensor that captures the face of a vehicle driver to generate a driver image. The driver monitor camera 3 has a two-dimensional detector configured with an array of photoelectric conversion elements, such as a CCD or C-MOS, that are sensitive to infrared light, and an imaging optical system that forms an image of the area to be captured on the two-dimensional detector. The driver monitor camera 3 also has a light source that emits infrared light. The driver monitor camera 3 is mounted, for example, at the front of the vehicle interior, facing the face of the driver seated in the driver's seat. The driver monitor camera 3 irradiates the driver with infrared light at a predetermined capture period (for example, 1 / 30 to 1 / 10 seconds) and outputs driver images in time series that include a facial area showing the driver's face.

[0019] The speaker device 4 is an example of an output device and includes, for example, an amplifier and a speaker unit. The speaker device 4 outputs a sound representing a message in accordance with a signal received from the notification device 8 via the in-vehicle network. The vehicle 1 may include a display device having a liquid crystal display as an output device. The output device may also be a light source that lights up in a pattern corresponding to the content of the message, a vibrator that vibrates in a pattern corresponding to the content of the message, or the like.

[0020] The GNSS receiver 5 is an example of a positioning sensor, which receives GNSS signals from GNSS satellites at predetermined intervals and, based on the received GNSS signals, determines the position of the vehicle 1. The GNSS receiver 5 outputs, at predetermined intervals, a positioning signal representing the positioning result of the vehicle 1's position based on the GNSS signals to the cruise control device 7 via the in-vehicle network.

[0021] The storage device 6 is an example of a storage unit, and includes, for example, a hard disk drive or a non-volatile semiconductor memory. The storage device 6 stores a high-precision map. The high-precision map includes, for example, information representing lane markings that separate lanes in a predetermined area depicted on the high-precision map.

[0022] The cruise control device 7 is an ECU (Electronic Control Unit) equipped with a communication interface, a memory, and a processor. The cruise control device 7 reads information about lane markings around the vehicle's position, which is represented in the positioning signal received from the GNSS receiver 5, from the storage device 6 that stores a high-precision map. The cruise control device 7 detects surrounding lane markings from the surrounding image received from the surrounding camera 2 and identifies the lane in which the vehicle 1 is traveling by matching the detected lane markings with the lane markings on the high-precision map. The cruise control device 7 detects surrounding objects, such as other vehicles, from the surrounding image received from the surrounding camera 2 and creates a travel route that maintains a predetermined distance from the objects and maintains or changes the lane depending on the situation. The cruise control device 7 then outputs a control signal to a driving mechanism (not shown) of the vehicle 1 so that the vehicle travels along the travel route. The surrounding camera 2, the GNSS receiver 5, the storage device 6, and the cruise control device 7, which work together to achieve autonomous driving, are an example of an autonomous driving system. The driving mechanism includes, for example, an engine or motor that supplies power to the vehicle 1, an accelerator for adjusting the acceleration of the vehicle 1, a brake for reducing the driving speed of the vehicle 1, and a steering mechanism for steering the vehicle 1.

[0023] The driving control device 7 can control the driving of the vehicle 1 at an autonomous driving level (autonomous driving level 3) where the driver is not required to monitor the surroundings under specified conditions, such as when the vehicle speed is within a specified range and the behavior of surrounding traffic participants is not complex. While the driving of the vehicle 1 is controlled by autonomous driving level 3, the driver can perform tasks other than driving (for example, operating a smartphone or relaxing).

[0024] When the conditions for driving control at Level 3 autonomous driving are no longer met, the driving control device 7 requests the driver to take over at least some of the tasks related to driving the vehicle 1, such as monitoring the surrounding conditions and controlling the driving mechanisms. At this time, if the driver is in a state where they cannot quickly take over the tasks, it becomes difficult for the vehicle 1 to drive safely. Therefore, the driving control device 7 detects the driver's state from the driver image received from the driver monitor camera 3, and if the driver is in a state where they cannot quickly take over the tasks, such as falling asleep or leaving their seat, the driving control device 7 issues a warning to the driver via the speaker device 4. In addition, if the driving control device 7 cannot detect the driver's state from the driver image, such as when the driver's face is hidden from the driver monitor camera 3 or the driver's eyes cannot be detected for a predetermined period of time, it is unclear whether the driver is in a state where they can take over the tasks, and so it issues a warning to the driver via the speaker device 4.

[0025] If the driving control device 7 is still unable to confirm that the driver is in a state where he or she can take over the task even after receiving the warning, the driving control device 7 decelerates the vehicle 1 and stops it in a safe place such as the shoulder of the road.

[0026] The notification device 8 is an ECU including a communication interface, a memory, and a processor. The notification device 8 notifies the driver of a message via the speaker device 4 based on the driver's behavior detected from the driver image received from the driver monitor camera 3. The notification device 8 may be implemented in the same ECU in which the driving control device 7 is implemented.

[0027] 2 is a hardware schematic diagram of the notification device 8. The notification device 8 includes a communication interface 81, a memory 82, and a processor 83.

[0028] The communication interface 81 is an example of a communication unit, and has a communication interface circuit for connecting the notification device 8 to an in-vehicle network. The communication interface 81 supplies the received data to the processor 83. The communication interface 81 also outputs the data supplied from the processor 83 to the outside.

[0029] The memory 82 is an example of a storage unit and includes a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 82 stores various data used in processing by the processor 83, such as a behavior detection table that associates the driver's state detected from the driver image with predetermined behaviors of the driver, a control termination behavior table that associates each predetermined behavior detected from the driver image with a control termination behavior that may be performed following the behavior, and a message table that associates each control termination behavior with a message to suppress the control termination behavior. The memory 82 also stores various application programs, such as a notification program that executes notification processing.

[0030] 3 is a diagram for explaining an outline of the behavior detection table. The memory 82 stores the behavior detection table, which associates the content detected from the driver image with predetermined behavior of the driver.

[0031] In the behavior detection table 821, for example, the behavior "gazing at terminal" is associated with the detection content "terminal" and "face direction (dashboard)." This indicates that the behavior "gazing at terminal" should be detected when a terminal (smartphone, tablet terminal, etc.) is detected from the driver image and the driver's face in the driver image is detected to be facing the dashboard.

[0032] In addition, in the behavior detection table 821, for example, the behavior "looking around inside the vehicle" is associated with the detection content "face direction (inside the vehicle)." This indicates that the behavior "looking around inside the vehicle" should be detected when it is detected from the driver image that the driver is turning his / her face toward a position inside the vehicle, such as the passenger seat / rear seat, for a predetermined period of time (for example, 5 seconds) or more.

[0033] Furthermore, in the behavior detection table 821, for example, the behavior "relax" is associated with the detection content "recline seat." This indicates that the behavior "relax" should be detected when an action such as the driver moving the driver's seat backward or reclining the backrest of the driver's seat is detected from the driver image. Instead of detecting an action such as the driver moving the driver's seat backward or reclining the backrest of the driver's seat, the behavior "relax" may be detected when it is detected that the position of the driver's head is further backward than usual.

[0034] 4 is a diagram for explaining an outline of the control end action table. The memory 82 stores the control end action table, which associates each predetermined action detected from the driver image with a corresponding control end action.

[0035] In the control termination action table 822, for example, the action "gazing at terminal" is associated with the control termination action "blocking camera." This indicates that when the action "gazing at terminal" is detected from the driver image, the control termination action "blocking camera" may be performed subsequently. The control termination action "blocking camera" corresponds to a situation in which the driver's face is not displayed in the driver image because the driver's face is out of the field of view of the driver monitor camera 3 or an obstacle is located between the driver's face and the driver monitor camera 3. The control termination action "blocking camera" causes the driver monitor camera 3 to be unable to properly generate a driver image including the face area, leading to a situation in which the driver's condition cannot be determined, and therefore is set as a condition for terminating control by the driving control device 7.

[0036] Furthermore, in the control termination action table 822, for example, the action "Look around the vehicle interior" is associated with the control termination actions "Face Lost" and "Leave Seat." This indicates that when the action "Look around the vehicle interior" is detected from the driver image, the control termination actions "Face Lost" or "Leave Seat" may be subsequently performed. The control termination action "Face Lost" corresponds to a situation in which the driver's eyes cannot be detected from the driver image because the driver is not facing the driver monitor camera 3. The control termination action "Face Lost" causes the driver monitor camera 3 to be unable to properly generate a driver image including a facial area, leading to a situation in which the driver's condition cannot be determined, and is therefore set as a termination condition for control by the driving control device 7. The control termination action "Leave Seat" corresponds to a situation in which the driver leaves the driver's seat. The control termination action "Leave Seat" causes a situation in which the driver is unable to properly take over tasks related to driving the vehicle 1, and is therefore set as a termination condition for control by the driving control device 7.

[0037] In the control termination action table 822, for example, the action "Relax" is associated with the control termination actions "Doze" and "Lost Face," and is associated with "a" as the control termination action "Driving Environment." This indicates that when the action "Relax" is detected from the driver image, the action corresponding to "Doze," "Lost Face," or "Driving Environment" a may be subsequently performed. The control termination action "Doze" indicates that the driver's alertness level drops below a predetermined threshold while seated in the driver's seat, which may lead to a situation in which the driver is unable to properly take over tasks related to driving the vehicle 1, and is therefore set as a condition for terminating control by the cruise control device 7. The control termination action "Driving Environment" indicates that the driving environment may be such that the driver is unable to properly take over tasks related to driving the vehicle 1 based on a predetermined driver behavior, and is also set as a condition for terminating control by the cruise control device 7. In the control termination action table 822, the types of driving environments that may result from predetermined actions are represented by symbols. The symbol "a" corresponds to "seat movement" in the control termination behavior "driving environment," i.e., the driver moving the driver's seat backward or reclining the driver's seat back. Such a driving environment tends to make the driver relaxed, which may lead to a situation where the driver is unable to properly take over the tasks related to driving the vehicle 1. Other types of driving environment are represented in the control termination behavior "driving environment," such as symbol "b" (placing an object around the driver's seat), symbol "c" (the driver changing their posture), "d" (the driver removing their glasses), and symbol "e" (the driver taking off their shoes).

[0038] 5 is a diagram for explaining an outline of the message table. The memory 82 stores a message table that associates control termination actions with messages for suppressing the control termination actions.

[0039] For example, in the message table 823, the control end action "cover the camera" is associated with audio data for causing the speaker device 4 to reproduce the message "Please be careful not to cover the camera inside the vehicle.".

[0040] Furthermore, in the message table 823, the control termination action "driving environment" is associated with audio data for playing a voice message saying, "Due to sudden changes in the driving environment, etc., you may be asked to take over driving in 10 seconds." The control termination action "driving environment" is further associated with audio data for playing a voice message corresponding to a code representing the type of driving environment. With this association, for example, for type "a" in the control termination action "driving environment," the message "Due to sudden changes in the driving environment, you may be asked to take over driving in 10 seconds" is notified, followed by the message "Please do not lower or recline the seat too much," allowing the driver to specifically understand the behavior that the message targets.

[0041] In the message table 823, each message is associated with a probability that the corresponding control termination action will be suppressed when the message is notified to the driver. For example, type "a" in the control termination action "driving environment" is associated with a probability "0.6" that the type "a" in the control termination action "driving environment" will be suppressed when the message "Due to sudden changes in the driving environment, etc., you may be asked to take over driving in 10 seconds" is notified followed by the message "Please do not lower or recline the seat too much."

[0042] The probability associated with each message is set based on an actual measurement value obtained by notifying a driver who is taking a predetermined action of the message and measuring whether the driver takes the expected control termination action. Alternatively, the probability associated with each message may be set based on an expected value that notifying a driver who is taking a predetermined action of the message is expected to suppress the control termination action.

[0043] Each message may be associated with a probability that a control termination action will be performed if the driver is not notified of the message, instead of or in addition to the probability that the corresponding control termination action will be suppressed if the driver is notified of the message.

[0044] The messages in the message table 823 may be text data for displaying predetermined characters on a display device (not shown).

[0045] The processor 83 is an example of a control unit and includes one or more processors and their peripheral circuits. The processor 83 may further include other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit.

[0046] FIG. 3 is a functional block diagram of the processor 83 included in the notification device 8. As shown in FIG.

[0047] The processor 83 of the notification device 8 has, as functional blocks, a detection unit 831, an identification unit 832, and a notification unit 833. Each of these units in the processor 83 is a functional module implemented by a computer program stored in the memory 82 and executed on the processor 83. The computer program that realizes the functions of each unit in the processor 83 may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. Alternatively, each of these units in the processor 83 may be implemented in the notification device 8 as an independent integrated circuit, microprocessor, or firmware.

[0048] The detection unit 831 detects a predetermined behavior of the driver from a driver image including a face area showing the face of the driver of the vehicle 1. The detection unit 831 also outputs a certainty factor indicating the likelihood of the detection.

[0049] The detection unit 831 detects the positions of predetermined parts of the driver, such as the driver's hands and head, the direction of the driver's face, and the types and positions of objects present around the driver's seat, by inputting each of the driver images acquired in time series from the driver monitor camera 3 into a classifier. Then, the detection unit 831 detects the driver's actions by identifying the driver's behavior and the objects.

[0050] The classifier that detects the positions of the driver's specific body parts, the driver's facial orientation, and the type and location of objects around the driver's seat can be, for example, a convolutional neural network (CNN) with multiple convolutional layers connected in series from the input side to the output side. By using images containing at least one of the driver's specific body parts, the driver's facial feature points, or an object as training data and training the CNN in advance according to a predetermined learning method such as backpropagation, the CNN operates as a classifier that identifies the positions of the driver's specific body parts, the positions of the driver's facial feature points, and the type and location of objects around the driver's seat. The detection unit 831 compares the detected positions of the driver's facial feature points with a standard 3D face model and detects the facial orientation in the 3D model when the positions of the facial feature points best match the positions detected in the image as the driver's facial orientation depicted in the image. Alternatively, the classifier can be a support vector machine (SVM). The SVM operates as a classifier that detects the specific body parts by defining support vectors that separate areas that include the specific body parts from areas that do not.

[0051] The detection unit 831 refers to the behavior detection table stored in the memory 82 and detects behavior corresponding to the detection content from the driver image. For example, if a terminal is detected from the driver image and it is detected that the driver is looking at the dashboard, the detection unit 831 detects the behavior "gazing at terminal" as the corresponding behavior. Also, for example, if a book is detected from the driver image and it is detected that the driver is looking at his / her hand, the detection unit 831 detects the behavior "reading."

[0052] Furthermore, the detection unit 831 refers to a behavior detection table stored in the memory 82 and detects behavior based on the detection content from each of a series of driver images acquired in time series. For example, the detection unit 831 detects the driver's facial direction from each of a series of driver images. If an image in which the facial direction is detected within a range corresponding to a position inside the vehicle, such as the passenger seat / rear seat, continues for a predetermined time or more, the detection unit 831 detects the behavior "looking around inside the vehicle."

[0053] The classifier outputs a certainty factor indicating the likelihood of the detection result for each of the predetermined body parts of the driver, the facial feature points of the driver, and the objects present around the driver's seat detected from the image. The detection unit 831 calculates the certainty factor of the detection of the behavior by, for example, multiplying the certainty factors for each of the detection contents from the driver image used for behavior detection (the predetermined body parts of the driver, the facial feature points of the driver, the objects present around the driver's seat, and the driver's facial direction, etc.). The detection unit 831 may calculate the average, maximum, or minimum value of the certainty factors for the detection contents from the driver image used for behavior detection as the certainty factor of the detection of the behavior.

[0054] The detection unit 831 may be further configured to detect a control termination behavior of the driver from the driver image. For example, if the size of an area of ​​consecutive pixels whose brightness difference from surrounding pixels is equal to or less than a predetermined brightness threshold is equal to or greater than a predetermined size threshold, the detection unit 831 detects the area as an obstruction area where the camera is obstructed by some kind of obstruction. Then, if a predetermined number of pixels (e.g., 1 / 3 or more of all pixels) in the driver image are obstruction areas, the detection unit 831 detects the control termination behavior "camera obstruction." Furthermore, if the driver's face is not detected from the driver image in detection using a classifier, for example, the detection unit 831 detects the control termination behavior "face lost."

[0055] The identification unit 832 refers to the control termination action table stored in the memory 82 and identifies a control termination action that may be performed following an action detected from the driver image. For example, if the action "gazing at the terminal" is detected from the driver image, the identification unit 832 identifies "blocking the camera" as a control termination action that may be performed following that action.

[0056] Furthermore, the identification unit 832 refers to the message table stored in the memory 82 and identifies a message for suppressing a control termination action corresponding to the action detected from the driver image. For example, if the action "stop terminal" is detected from the driver image, the identification unit 832 identifies a message "be careful not to cover the in-vehicle camera" for identifying the control termination action "block the camera."

[0057] The notification unit 833 calculates the effectiveness level by multiplying the confidence level in the detection of the predetermined behavior by the probability that the control termination behavior will be suppressed when the message is notified to the driver. Then, if the calculated effectiveness level is greater than the predetermined notification threshold value stored in the memory 82, the notification unit 833 notifies the driver of the identified message via the speaker device 4.

[0058] When a probability that a control terminating action will be performed if the message is not notified to the driver is associated with each message in the message table stored in memory 82, notification unit 833 may calculate the effectiveness level by multiplying the probability by the confidence level in detecting the predetermined action. In this case, in determining the effectiveness level, notification unit 833 may use a notification threshold value that is different from the notification threshold value used to determine the effectiveness level calculated based on the probability that the control terminating action will be suppressed if the message is notified to the driver.

[0059] When the notification unit 833 notifies the driver of a message, the notification unit 833 may store information identifying the message and the time of notification in the memory 82. Before notifying the identified message, the notification unit 833 calculates the elapsed time from the time when the message was most recently notified to the current time. Then, if the elapsed time is shorter than a predetermined message interval threshold (e.g., 60 seconds) stored in the memory 82, the notification unit 833 suppresses (does not execute) the notification of the identified message. By operating the notification unit 833 in this manner, the notification device 8 can reduce the annoyance felt by the driver due to repeated notifications of the same message in a short period of time.

[0060] Furthermore, if the time elapsed from the most recent notification of a message different from the identified message to the current time is shorter than the different message interval threshold, the notification unit 833 may suppress notification of the identified message. The different message interval threshold may be set to a value equal to or less than the message interval threshold (for example, 30 seconds).

[0061] When the detection unit 831 is configured to detect the driver's control termination action from the driver image, the notification unit 833 may notify the driver of a notice of the end of driving control by the driving control device 7 via the speaker device 4 in response to the detection of the control termination action. At this time, the notice of the end of control by the driving control device 7 may be notified in a manner that is stronger in intensity (for example, at a louder volume in the case of notification by the speaker device 4) than the notice of a message for suppressing the control termination action. By operating the notification unit 833 in this manner, the notification device 8 can properly notify the driver of the notice of the end of driving control.

[0062] Furthermore, the detection unit 831 may suppress notification of the identified message if the elapsed time since the most recent cruise control termination notice based on the detection of the control termination action corresponding to the identified message is shorter than a predetermined termination notice interval threshold. The termination notice interval threshold may be set to a value greater than the message interval threshold (e.g., 180 seconds). By operating the notification unit 833 in this manner, the notification device 8 can reduce the annoyance to the driver caused by receiving messages related to cruise control termination notices that have already been notified at short intervals.

[0063] Furthermore, if the time elapsed from the most recent notification of the end of driving control based on the detection of a control end action different from the control end action corresponding to the identified message to the current time is shorter than the separate end notice interval threshold, the notification unit 833 may suppress notification of the identified message. The separate end notice interval threshold may be set to a value equal to or less than the end notice interval threshold (for example, 30 seconds).

[0064] 7 is a flowchart of the notification process. The notification device 8 repeatedly executes the process shown in FIG.

[0065] First, the detection unit 831 of the processor 83 of the notification device 8 detects a predetermined behavior of the driver from a driver image including the facial area of ​​the driver of the vehicle 1 generated by the driver monitor camera 3 (step S1), and if no behavior is detected (step S1: N), the notification process is terminated.

[0066] If a predetermined behavior of the driver is detected from the driver image (step S1: Y), the detection unit 831 outputs the degree of certainty of the detection (step S2).

[0067] Next, the specifying unit 832 of the processor 83 of the notification device 8 specifies a message for suppressing a control termination behavior that may be performed following the detected behavior (step S3).

[0068] Next, the notification unit 833 of the processor 83 of the notification device 8 calculates an effectiveness level by multiplying the confidence level in the detection of the behavior by a predetermined probability that the control termination behavior will be suppressed when the specified message is notified to the driver. Then, the notification unit 833 determines whether the calculated effectiveness level is greater than the notification threshold value stored in the memory 82 (step S4).

[0069] If it is determined that the calculated validity level is greater than the notification threshold value stored in memory 82 (step S4: Y), the notification unit 833 notifies the driver of the identified message via the speaker device 4 (step S5) and ends the notification process. On the other hand, if it is determined that the calculated validity level is not greater than the notification threshold value stored in memory 82 (step S4: N), the notification unit 833 ends the notification process without notifying the driver of the identified message.

[0070] By executing the notification process in this manner, the notification device 8 can properly notify the driver of the vehicle of a message so that the driver can continue autonomous driving.

[0071] In a modified example, the detection unit 831 determines whether a control terminating action is detected after notifying the driver of a message through the notification process (step S4: Y → step S5).The detection unit 831 then calculates the probability that the control terminating action would be suppressed if a message were notified by dividing the number of times that the control terminating action was not detected after notifying the driver of a message through the notification process by the number of times that the message was notified.The detection unit 831 then updates the probability stored in association with each message in the message table 823 saved in the memory 82 based on the calculated probability.

[0072] Furthermore, the detection unit 831 may determine whether or not a control termination action is detected after the notification process has not notified the driver of a message (step S4: N → End). In this case, the detection unit 831 calculates the probability that the control termination action will be performed when no message is notified by dividing the number of times that the control termination action was detected after the notification process has not notified the driver of a message by the number of times that the message was notified, and updates the probability in the message table 823 with the calculated probability.

[0073] In such a modified example, the message can be notified in a manner that matches the driver's tendency to respond to messages.

[0074] In another modification, the detection unit 831 detects a control termination action from the driver image and determines the frequency (for example, once every 100 hours) at which the control termination action is detected while driving control is being performed by the driving control device 7. Then, if the determined frequency is smaller than the frequency threshold stored in the memory 82 (for example, the frequency threshold is once every 80 hours and the determined frequency is once every 100 hours), the detection unit 831 may update the notification threshold to a value smaller than the notification threshold stored in the memory 82.

[0075] It is believed that a driver whose frequency of detecting control termination actions has decreased has a greater understanding of actions that lead to the termination of automated driving control. In this modified example, the notification of a message to such a driver with a greater understanding is limited to cases where the effectiveness level is higher, thereby reducing the annoyance experienced by the driver.

[0076] It should be understood that those skilled in the art can make various changes, substitutions and alterations thereto without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0077] 1 vehicle 2 Peripheral Camera 3 Driver monitor camera 4. Speaker equipment 5 GNSS receivers 6 Storage devices 7. Driving control device 8 Notification device 831 Detector 832 Specific part 833 Notification Department

Claims

1. a detection unit that detects a predetermined behavior of a driver of a vehicle whose driving is controlled by an automated driving system from a driver image including a face area representing the face of the driver, and outputs a certainty factor indicating the likelihood of the detection; an identification unit that identifies a message for suppressing a control termination action that may be performed following the detected predetermined action and that corresponds to a termination condition for control by the autonomous driving system; a notification unit that notifies the driver of the specified message via an output device when an effectiveness level calculated by multiplying the certainty factor by a first probability that the control termination action will be suppressed if the specified message is notified to the driver, or by a second probability that the control termination action will be performed if the specified message is not notified to the driver, is greater than a predetermined notification threshold; and A notification device comprising:

2. The notification device according to claim 1 , wherein the notification unit suppresses notification of the specified message if the elapsed time since the specified message was last notified is shorter than a predetermined message interval threshold.

3. The detection unit further detects the control termination action of the driver from the driver image, The notification device according to claim 2 , wherein the notification unit notifies the driver of an advance notice of the end of control by the autonomous driving system via the output device in a manner that is stronger than the message notification in response to the detection of the control termination action.

4. The notification device described in claim 3, wherein the notification unit suppresses notification of the identified message if the elapsed time from the time when the termination notice based on the detection of the control termination action corresponding to the identified message was most recently notified is shorter than a predetermined termination notice interval threshold that is longer than the message interval threshold.

5. detecting a predetermined behavior of a driver of a vehicle whose driving is controlled by an automated driving system from a driver image including a face region representing the face of the driver, and outputting a confidence level indicating the likelihood of the detection; Identifying a message for suppressing a control termination action that may be performed following the detected predetermined action and that corresponds to a termination condition for control by the automated driving system; notifying the driver of the specified message via an output device when an effectiveness level calculated by multiplying the confidence by a first probability that the control termination action will be suppressed if the specified message is notified to the driver, or by a second probability that the control termination action will be performed if the specified message is not notified to the driver, is greater than a predetermined notification threshold; Notification methods, including:

6. detecting a predetermined behavior of a driver of a vehicle whose driving is controlled by an automated driving system from an image including a face region representing the face of the driver, and outputting a confidence level indicating the likelihood of the detection; Identifying a message for suppressing a control termination action that may be performed following the detected predetermined action and that corresponds to a termination condition for control by the automated driving system; notifying the driver of the specified message via an output device when an effectiveness level calculated by multiplying the confidence by a first probability that the control termination action will be suppressed if the specified message is notified to the driver, or by a second probability that the control termination action will be performed if the specified message is not notified to the driver, is greater than a predetermined notification threshold; A notification computer program that causes a computer to do something.

Images