Message notification device and message notification method

Abstract

Provided are a message notification device and a message notification method that can appropriately notify a driver of a message even in a vehicle controlled to travel in an automatic driving mode. The message notification device displays a message that the driver should be aware of in a display portion disposed in front of a driver seat on which the driver sits, when the vehicle is controlled to travel in an automatic driving mode in which a driving operation of the driver is not required, and a line-of-sight direction of the driver is not directed toward a direction related to the driving operation, in a case where a change in a surrounding environment of the vehicle to the extent that the driver directs his or her attention to the front is detected, a behavior of the vehicle to the extent that the driver can feel is detected, or an operation of a vehicle-mounted device mounted on the front of the vehicle by the driver is detected.

Description

Technical Field

[0001] This disclosure relates to a message notification device for notifying drivers of vehicles. Background Technology

[0002] A message notification device mounted on a vehicle notifies the driver of various messages. Patent Document 1 describes a head-up display device that determines the display content or position based on the driver's attribute information and projects image light onto a projection component positioned in front of the driver to display a virtual image.

[0003] Existing technical documents

[0004] Patent Document 1: Japanese Patent Application Publication No. 2019-059245 Summary of the Invention

[0005] The technical problem that the invention aims to solve

[0006] In vehicles operating in autonomous driving mode, which does not require driver intervention, particularly Level 3 autonomous driving where the driver has no obligation to monitor the surroundings, the driver is not required to look ahead. Therefore, even if a message is displayed on a monitor positioned in front of the driver, the driver may not be able to properly recognize it.

[0007] The purpose of this disclosure is to provide a message notification device that can appropriately notify the driver of messages even in a vehicle controlled in autonomous driving mode.

[0008] Technical solutions for solving the problem

[0009] The message notification device disclosed herein includes: a gaze detection unit that detects the gaze direction of the driver of the vehicle; and a notification unit that, when controlling the vehicle in an autonomous driving mode that does not require the driver's driving operation and the driver's gaze direction is not towards a direction related to driving operation, displays a message that should be known to the driver on a display unit located in front of the driver's seat, when it detects changes in the vehicle's surrounding environment to the extent that the driver is paying attention to the road ahead, changes in the vehicle's behavior to the extent that the driver can perceive, or operations by the driver on an in-vehicle device mounted in front of the driver's seat.

[0010] The notification device disclosed herein preferably further includes a determination unit that determines the message that should be communicated to the driver based on changes in the environment inside the vehicle or changes in the surrounding environment of the vehicle to the extent that the driver has not paid attention to what is ahead.

[0011] In the message notification device disclosed herein, it is preferable that the notification unit further detects, within a predetermined time after detecting changes in the surrounding environment, changes in vehicle behavior, or driver operation, that the driver's gaze is directed in a direction related to driving operation, and then displays a message on the display unit.

[0012] In the message notification device of this disclosure, preferably, changes in the surrounding environment include at least one of changes in the illuminance around the vehicle, changes in the air pressure around the vehicle, and changes in the interval from the vehicle to a preceding vehicle traveling in front of the vehicle.

[0013] In the notification device disclosed herein, preferred changes in vehicle behavior include at least one of the following: starting or stopping the operation of the windshield wipers, turning on or off the headlights, and stopping or slowing down the vehicle.

[0014] In the message notification device disclosed herein, the operation of the preferred in-vehicle equipment includes the operation of at least one of a vehicle navigation device, an audio device, and an air conditioning device.

[0015] The notification method disclosed herein includes: when controlling the vehicle in an autonomous driving mode that does not require driver operation and the driver's gaze is not directed in a direction related to driving operation, displaying a message that should be known to the driver on a display unit located in front of the driver's seat when the following are detected: changes in the vehicle's surrounding environment to the extent that the driver's attention is focused forward; changes in the vehicle's behavior to the extent that the driver can perceive; or operations by the driver on an in-vehicle device mounted in front of the driver's seat.

[0016] The message notification method disclosed herein preferably further includes: determining a message that the driver should be informed of based on changes in the environment inside the vehicle or changes in the surrounding environment of the vehicle to the extent that the driver has not focused their attention on the road ahead.

[0017] The message notification method disclosed herein preferably displays a message on the display unit when, within a predetermined time after detecting a change in the surrounding environment, a change in vehicle behavior, or a driver's operation, it is further detected that the driver's gaze is directed in a direction related to the driving operation.

[0018] In the notification method disclosed herein, preferred changes in the surrounding environment include at least one of changes in illuminance around the vehicle, changes in air pressure around the vehicle, and changes in the interval from the vehicle to a preceding vehicle traveling in front of the vehicle.

[0019] In the notification method disclosed herein, preferred changes in vehicle behavior include at least one of the following: starting or stopping the operation of the windshield wipers, turning on or off the headlights, and stopping or slowing down the vehicle.

[0020] In the notification method disclosed herein, the operation of the in-vehicle equipment preferably includes the operation of at least one of the vehicle navigation device, audio equipment, and air conditioning equipment.

[0021] The information disclosed herein uses a computer program to cause the computer installed in the vehicle to execute the following: when controlling the vehicle's driving in an automated driving mode that does not require driver operation and the driver's gaze is not directed in a direction related to driving operation, the program will display a message that should be known to the driver on a display unit located in front of the driver's seat if it detects changes in the vehicle's surrounding environment to the extent that the driver's attention is directed forward, changes in the vehicle's behavior to the extent that the driver can perceive, or operations by the driver on an in-vehicle device located in front of the driver's seat.

[0022] The computer program for disclosing this information preferably further enables the computer to perform the following: determine the information that should be communicated to the driver based on changes in the environment inside the vehicle or changes in the surrounding environment of the vehicle to the extent that the driver has not focused their attention on the road ahead.

[0023] In the computer program for message notification disclosed herein, it is preferable that, within a predetermined time after detecting a change in the surrounding environment, a change in vehicle behavior, or a driver's operation, the display unit displays a message.

[0024] In the computer program for notification of information disclosed herein, preferred changes in the surrounding environment include at least one of changes in illuminance around the vehicle, changes in air pressure around the vehicle, and changes in the interval from the vehicle to a preceding vehicle traveling in front of the vehicle.

[0025] In the computer program for notification of information disclosed herein, the preferred changes in vehicle behavior include at least one of the following: starting or stopping the operation of the windshield wipers, turning on or off the headlights, and stopping or slowing down the vehicle.

[0026] In the computer program for notification of information disclosed herein, the operation of preferred in-vehicle equipment includes the operation of at least one of a vehicle navigation device, an audio device, and an air conditioning device.

[0027] According to the message notification device disclosed herein, even in a vehicle controlled in autonomous driving mode, it is possible to appropriately notify the driver of messages. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a vehicle equipped with a notification device.

[0029] Figure 2 This is a hardware diagram of the ECU.

[0030] Figure 3 This is a functional block diagram of the processor in an ECU.

[0031] Figure 4 This is a flowchart of message notification processing.

[0032] Label Explanation

[0033] 1 vehicle

[0034] 8ECU

[0035] 831 Line of Sight Inspection Department

[0036] 837 Notification Department Detailed Implementation

[0037] The following description, with reference to the accompanying drawings, details a message notification device that can appropriately notify the driver even when the vehicle is being driven in autonomous driving mode. The message notification device determines whether the vehicle is being driven in autonomous driving mode, where driver input is not required. Furthermore, the message notification device determines whether the driver's gaze is directed in a direction related to driving operations. When the vehicle is being driven in autonomous driving mode and the driver's gaze is not directed in a direction related to driving operations, the message notification device further determines whether it detects changes in the vehicle's surrounding environment to the extent that the driver is focusing their attention forward, whether it detects changes in the vehicle's behavior to the extent that the driver can perceive them, or whether it detects the driver's operation of an in-vehicle device mounted at the front of the vehicle. Furthermore, if the message notification device detects changes in the surrounding environment, changes in the vehicle's behavior, or driver operations, it displays a message that the driver should be aware of on a display unit located in front of the driver's seat.

[0038] Figure 1 This is a schematic diagram of a vehicle equipped with a notification device.

[0039] Vehicle 1 includes a perimeter camera 2, a driver monitoring camera 3, a GNSS (Global Navigation Satellite System) receiver 4, a storage device 5, in-vehicle equipment 6, an instrument display 7, and an ECU 8. ECU 8 is an example of a message notification device. The perimeter camera 2, driver monitoring camera 3, GNSS receiver 4, storage device 5, in-vehicle equipment 6, instrument display 7, and ECU 8 are connected in a communicable manner via an in-vehicle network conforming to a standard controller area network.

[0040] The peripheral camera 2 is an example of a peripheral imaging unit for generating peripheral images representing the vehicle's surrounding environment. The peripheral camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as CCD or C-MOS, and an imaging optical system that images the area to be photographed onto the two-dimensional detector. The peripheral camera 2 is, for example, positioned forward-facing in the upper front part of the vehicle interior, and captures images of the vehicle's surroundings through the windshield at predetermined shooting intervals (e.g., 1 / 30 to 1 / 10 of a second), outputting a peripheral image representing the surrounding conditions.

[0041] The driver monitoring camera 3 is an example of a driver imaging unit used to generate a driver image showing the state of the driver's head in a vehicle. The driver monitoring camera 3 consists of a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared light, such as CCD or C-MOS, and an imaging optical system that images the area to be photographed onto the two-dimensional detector. Furthermore, the driver monitoring camera 3 has a light source that emits infrared light. The driver monitoring camera 3 is mounted, for example, at the front of the vehicle interior facing the face of the driver seated in the driver's seat. The driver monitoring camera 3 illuminates the driver with infrared light at predetermined shooting intervals (e.g., 1 / 30 to 1 / 10 of a second), and outputs an image showing the driver's face.

[0042] GNSS receiver 4 receives GNSS signals from GNSS satellites at predetermined intervals and uses the received GNSS signals to determine the position of vehicle 1. GNSS receiver 4 outputs position measurement signals to ECU 8 and other devices via the in-vehicle network at predetermined intervals, and the position measurement signals represent the position measurement results of vehicle 1 based on the GNSS signals.

[0043] Storage device 5 is an example of a storage device, such as a hard disk drive or a non-volatile semiconductor memory. Storage device 5 stores map information, which includes paths between locations within a predetermined range.

[0044] The vehicle-mounted device 6 is a device installed in front of the driver's seat in the vehicle 1. The vehicle-mounted device 6 has a communication interface, a touch panel display, a memory storing map information, and a processor, and is an information processing device with navigation functions. Based on the vehicle 1's own position determined from the positioning signal obtained from the GNSS receiver 4, the vehicle-mounted device 6 searches for a path to the destination specified by operation of the touch panel display, and displays the searched path on the touch panel display.

[0045] The in-vehicle device 6 can be a device located in front of the driver's seat that accepts the driver's operation, or it can be a device with functions other than navigation, such as audio equipment, air conditioning equipment, etc.

[0046] The instrument display 7 is an example of a display unit, such as one with an LCD screen. The instrument display 7 is positioned in front of the driver's seat in the vehicle 1 and displays messages that the driver should be aware of, based on signals received from the ECU 8 via the in-vehicle network.

[0047] ECU8 is an ECU (Electronic Control Unit) with a communication interface, memory, and processor. Based on surrounding images received from the peripheral camera 2, ECU8 detects objects around the vehicle 1 and outputs control signals to the vehicle 1's engine or motor, brakes, steering system, and other driving mechanisms (not shown), executing automatic driving control of the vehicle 1 to maintain a predetermined distance from the detected objects. Additionally, ECU8 detects the driver's gaze direction based on images received from the driver monitoring camera 3 via the communication interface. Furthermore, ECU8 detects changes in the surrounding environment, changes in the vehicle 1's behavior, and the driver's operations on the onboard equipment 6. Finally, ECU8 displays information that should be known to the driver on the instrument display 7.

[0048] Figure 2 This is a hardware diagram of ECU8. ECU8 has a communication interface 81, a memory 82, and a processor 83.

[0049] Communication interface 81 is an example of a communication unit, having a communication interface circuit for connecting ECU 8 to the in-vehicle network. Communication interface 81 provides received data to processor 83. Additionally, communication interface 81 outputs data provided from processor 83 to an external source.

[0050] Memory 82 is an example of a storage unit, comprising both volatile and non-volatile semiconductor memory. Memory 82 stores various data used in the processing of processor 83, such as parameter sets for specifying the actions of a neural network that acts as a recognizer detecting objects from surrounding images, a template representing the corneal reflection of the pupil and light source for detecting the driver's gaze direction by matching it with a driver image, a reference for determining whether changes in the surrounding environment indicate a change in the driver's focus forward, and a reference for determining whether the driver can perceive changes in vehicle behavior. Additionally, memory 82 stores various application programs, such as message notification programs that execute message notification processing.

[0051] Processor 83 is an example of a control unit, having one or more processors and their peripheral circuitry. Processor 83 may also have other arithmetic circuitry such as logic operation units, numerical operation units, or graphics processing units.

[0052] Figure 3 This is a functional block diagram of the processor 83 in ECU8.

[0053] The processor 83 of the ECU 8 includes, as a functional block, a gaze detection unit 831, a peripheral detection unit 832, a driving control unit 833, a behavior detection unit 834, an operation detection unit 835, a determination unit 836, and a notification unit 837. These units of the processor 83 are functional modules installed via programs executed on the processor 83. The computer program implementing the functions of each unit of the processor 83 may also be provided in the form of a computer-readable, removable recording medium such as a semiconductor memory, magnetic recording medium, or optical recording medium. Alternatively, these units of the processor 83 may also be installed in the ECU 8 as independent integrated circuits, microprocessors, or firmware.

[0054] The gaze detection unit 831 detects the driver's gaze direction from the driver image received from the driver monitoring camera 3 via the communication interface 81. The gaze detection unit 831 detects the corneal reflection of the pupil and light source, for example, by performing template matching between a template representing the corneal reflection of the pupil and light source and the driver image, and detects the gaze direction based on their positional relationship. The gaze direction is represented by the horizontal and vertical angles between the vehicle 1's direction of travel and the driver's gaze.

[0055] Changes in the surrounding environment of vehicle 1, part of the surrounding inspection department 832.

[0056] The perimeter detection unit 832 detects the interval from vehicle 1 to the preceding vehicle traveling in front of vehicle 1 as the surrounding environment of vehicle 1. In this case, the perimeter detection unit 832 inputs the surrounding image to a pre-learned first recognizer in order to detect objects, thereby obtaining the positions of other vehicles contained in the surrounding image. The first recognizer can be, for example, a convolutional neural network (CNN) having multiple convolutional layers connected in series from the input side to the output side. By pre-learning by inputting the surrounding image containing other vehicles into the CNN as teacher data, the CNN operates as a first recognizer for determining the positions of other vehicles.

[0057] The peripheral detection unit 832 detects the inter-vehicle distance from vehicle 1 to other vehicles by comparing the width of other vehicles in the peripheral image with the width of a standard vehicle pre-stored in the memory 82.

[0058] Furthermore, the peripheral detection unit 832 detects the illumination around the vehicle 1 to define the surrounding environment of the vehicle 1. In this case, the peripheral detection unit 832 inputs a peripheral image to a second recognizer that has been pre-learned to perform semantic segmentation, and classifies the multiple pixels contained in the peripheral image into any one of multiple classes representing objects corresponding to the pixels. The second recognizer can be configured as a CNN having multiple convolutional layers connected in series from multiple input sides to the output side, but without fully connected layers. By pre-learning the peripheral image, which assigns a classification of each pixel to a class such as vehicle, road, or empty, into the CNN as teacher data, the CNN operates as a second recognizer performing semantic segmentation.

[0059] The peripheral detection unit 832 uses the brightness of pixels classified as roads and the aperture value and exposure time in the capture of peripheral images to estimate the illuminance around vehicle 1.

[0060] In addition, the peripheral detection unit 832 may also use the following recognizer as the first recognizer and the second recognizer, which performs class classification of each pixel of the peripheral image and performs recognition of each object corresponding to the class classified for that pixel.

[0061] In addition, the surrounding detection unit 832 detects the air pressure around the vehicle 1 based on the air pressure information obtained from the air pressure sensor (not shown) installed in the vehicle 1 and connected to the in-vehicle network, and uses this as the surrounding environment of the vehicle 1.

[0062] The driving control unit 833 controls the driving of the vehicle 1 in an automated driving mode that does not require driver operation. The driving control unit 833 generates a driving trajectory where the distance to an object detected by the peripheral detection unit 832 from the peripheral image is maintained at or above a predetermined level, and outputs control signals to the driving mechanism (not shown) of the vehicle 1 so that the vehicle 1 drives along the driving trajectory. The driving mechanism includes, for example, an engine or motor that supplies power to the vehicle 1, a brake that reduces the speed of the vehicle 1, and a steering mechanism that steers the vehicle 1.

[0063] Behavior detection department 834 detected changes in the behavior of vehicle 1.

[0064] The behavior detection unit 834 detects changes in the behavior of the windshield wipers (not shown) mounted on the vehicle 1 and connected to the in-vehicle network, such as the start or stop of their operation, as changes in the behavior of the vehicle 1. To detect changes in the windshield wiper behavior, the behavior detection unit 834 acquires windshield wiper behavior signals indicating appropriate wiper actions. It sends windshield wiper behavior signals to the wipers based on the driver's operation of the wiper switch or raindrop signals output by a raindrop sensor (not shown) mounted on the vehicle 1.

[0065] Furthermore, the behavior detection unit 834 detects changes in behavior, such as the illumination or extinguishing of headlights (not shown), which are mounted on the vehicle 1 and connected to the in-vehicle network, as behavior changes of the vehicle 1. To detect these headlight behavior changes, the behavior detection unit 834 acquires a headlight behavior signal indicating the headlight's behavior. The headlight behavior signal is sent to the headlights based on the driver's operation of the headlight switch or the illuminance signal output by an illuminance sensor (not shown) mounted on the vehicle 1.

[0066] Furthermore, the behavior detection unit 834 detects behaviors such as stopping or decelerating of the vehicle 1 as behaviors of the vehicle 1. To detect stopping or decelerating of the vehicle 1, the behavior detection unit 834 obtains a vehicle speed signal representing the vehicle speed from a vehicle speed sensor (not shown) mounted on the vehicle 1 and connected to the in-vehicle network. The vehicle speed of the vehicle 1 is changed based on the driver's operation of the brake pedal or control signals output by the driving control unit 833 to the driving mechanism.

[0067] The operation detection unit 835 detects the driver's operation of the in-vehicle device 6. The operation detection unit 835 detects the driver's operation of the in-vehicle device 6 by receiving, via the in-vehicle network, operation signals sent by the in-vehicle device 6 based on the driver's touch of the touch panel display of the in-vehicle device 6. Alternatively, the operation detection unit 835 can also detect the driver's operation of the in-vehicle device 6 by detecting the overlap between the in-vehicle device 6 and the driver's hand in an in-vehicle image output from an in-vehicle camera (not shown) configured to capture images of the in-vehicle device 6. In this case, the in-vehicle device 6 may not be connected to the in-vehicle network.

[0068] The determination unit 836 determines the information that should be communicated to the driver based on changes in the environment inside the vehicle 1 or changes in the surrounding environment of the vehicle 1 to the extent that the driver has not focused his attention on the road ahead.

[0069] The determination unit 836 receives temperature and humidity signals representing the temperature and humidity inside the vehicle from an interior temperature and humidity sensor (not shown) installed in the vehicle 1 and connected to the vehicle network. Furthermore, when a change in the interior environment corresponds to an interior environment change pattern stored in the memory 82, the determination unit 836 determines a message associated with that interior environment change pattern (e.g., "Please wipe the fog off the windshield").

[0070] Additionally, the determination unit 836 obtains traffic information indicating the traffic conditions ahead of its current position along the driving path from a traffic information receiver (not shown) that obtains traffic information via beacons installed on the road or FM multi-broadcast. Furthermore, if the change in the traffic conditions ahead corresponds to a traffic condition change pattern stored in the memory 82, the determination unit 836 determines a message associated with that traffic condition change pattern (e.g., "traffic congestion ahead"). Changes in the traffic conditions ahead are not directly perceptible to the driver; therefore, they represent changes in the surrounding environment due to the driver's lack of attention.

[0071] Furthermore, when the current position of vehicle 1, as indicated by the positioning signal obtained from GNSS receiver 4, approaches within a predetermined distance of the end point of autonomous driving, the determination unit 836 determines a message associated with the approach to the end point of autonomous driving (e.g., "Autonomous driving will end soon"). Since the driver cannot directly perceive the end point of autonomous driving, the approach to the end point of autonomous driving is considered a change in the surrounding environment where the driver has not focused their attention.

[0072] If all of the following conditions are met, the notification unit 837 displays the message that the determination unit 836 has determined should be known to the driver on the instrument display 7.

[0073] (Condition 1) Control the driving of vehicle 1 in autonomous driving mode.

[0074] The notification unit 837 determines whether it is controlling the driving of vehicle 1 in autonomous driving mode by questioning the driving control unit 833.

[0075] (Condition 2) The driver's gaze is not directed in a direction related to driving operations (eyes off)

[0076] The direction related to driving operation refers to the direction that the driver's gaze would be directed when performing driving operations. For example, the direction related to driving operation is within a range of ±15 degrees vertically and ±45 degrees horizontally relative to the direction of travel of vehicle 1, and is considered forward from the driver's perspective. The notification unit 837 obtains the driver's gaze direction from the gaze detection unit 831 and determines whether the driver is in a state of gaze deviance.

[0077] (Condition 3) At least one of the following conditions 3-1 to 3-3 is satisfied.

[0078] (Condition 3-1) A change in the surrounding environment was detected in the degree to which the driver focused their attention forward.

[0079] The memory 82 stores change thresholds according to the category of the surrounding environment, which are used to determine the degree to which the driver is focusing their attention forward. The notification unit 837 obtains values ​​such as illuminance, air pressure, and the interval to the preceding vehicle from the surrounding detection unit 832 in a time series as the surrounding environment. Furthermore, if the difference between the past value and the current value is greater than the change threshold, a change in the surrounding environment indicating the degree to which the driver is focusing their attention forward is detected. The past value can be the value at the most recent past time point in the time series, or it can be a representative value (e.g., an arithmetic mean) of values ​​over a predetermined period of the past that includes the most recent past time point.

[0080] (Condition 3-2) Detected a change in the behavior of vehicle 1 to a degree that the driver can perceive.

[0081] The memory 82 stores patterns of changes in the vehicle 1's behavior that can be perceived by the driver. The notification unit 837 obtains information about changes in the vehicle 1's behavior from the behavior detection unit 834. Furthermore, if the behavior change indicated by the obtained information belongs to any of the behavior change patterns stored in the memory 82, the degree to which the driver can perceive the change in the vehicle 1's behavior is detected.

[0082] (Condition 3-3) Driver's operation of vehicle-mounted device 6 was detected.

[0083] The notification department 837 obtains the driver's operation of the on-board equipment 6 as detected by the operation testing department 835.

[0084] Thus, the message determined by the determination unit 836 and displayed on the instrument display 7 by the notification unit 837 is not directly related to the events detected under the conditions 3-1 to 3-3 described above. The determination unit 836 is able to determine the message independently of the detection of these events by the notification unit 837.

[0085] Alternatively, if the notification unit 837 meets the following condition 4 based on the above condition 3, it may display a message on the instrument display 7.

[0086] (Condition 4) Within a predetermined time after Condition 3 is met, the driver's line of sight is detected to be directed in a direction related to driving operations.

[0087] Figure 4 This is a flowchart of the message notification process. During the driving of vehicle 1, ECU8 repeatedly performs message notification processing at predetermined time intervals (e.g., 1 / 10 second intervals).

[0088] First, the determination unit 836 of ECU8 determines the message that the driver should know (step S1).

[0089] Next, the notification unit 837 of ECU8 determines whether it is controlling the driving of vehicle 1 in autonomous driving mode (step S2). If it is determined that the driving of vehicle 1 is not being controlled in autonomous driving mode (step S2: no), ECU8 ends the message notification processing.

[0090] If it is determined that the vehicle 1 is being driven in autonomous driving mode (step S2: Yes), the notification unit 837 determines whether the driver is out of sight (step S3). If it is determined that the driver is not out of sight (step S3: No), the ECU 8 terminates the message notification processing.

[0091] If it is determined that the driver is out of sight (step S3: Yes), the notification unit 837 determines whether a change in the surrounding environment to the extent that the driver is turning their attention forward has been detected (step S4). If no change in the surrounding environment to the extent that the driver is turning their attention forward has been detected (step S4: No), the ECU 8 terminates the message notification processing.

[0092] Furthermore, if it is determined that the driver is out of sight (step S3: Yes), the notification unit 837 determines whether a change in the behavior of the vehicle 1 is detected to a degree that the driver can perceive (step S5). If no change in the behavior of the vehicle 1 is detected to a degree that the driver can perceive (step S5: No), the ECU 8 terminates the message notification processing.

[0093] Furthermore, if it is determined that the driver is out of sight (step S3: Yes), the notification unit 837 determines whether the driver's operation on the vehicle-mounted device 6 has been detected (step S6). If no operation on the vehicle-mounted device 6 has been detected (step S6: No), the ECU 8 terminates the message notification processing.

[0094] If a change in the surrounding environment is detected to the extent that the driver is focusing their attention ahead (step S4: Yes), a change in the behavior of the vehicle 1 is detected to the extent that the driver can perceive (step S5: Yes), or an operation of the vehicle-mounted device 6 by the driver is detected (step S6: Yes), the notification unit 837 displays a message that the driver should know on the instrument display 7 (step S7) and ends the message notification process.

[0095] By performing message notification processing in this way, ECU8 can also appropriately notify the driver of messages while controlling the vehicle in autonomous driving mode.

[0096] Furthermore, in message notification processing, it is not necessary to first perform the message determination (step S1) by the determination unit 836; it can be performed before the message is displayed (step S7).

[0097] Alternatively, during message notification processing, if it is determined that the driver is not out of sight (step S2: no), ECU8 can proceed to step S6, and notification unit 837 can display a message that the driver should know on the instrument display 7.

[0098] Alternatively, the in-vehicle device 6 may have a display, and in step S7 of the message notification processing, the notification unit 837 may cause the display of the in-vehicle device 6 to display a message that the driver should be aware of. That is, the in-vehicle device that detects the driver's operation and the display unit that displays the message that the driver should be aware of may not be separate units.

[0099] It is intended to be understood that those skilled in the art can make various changes, substitutions and modifications to this disclosure without departing from its spirit and scope.

Claims

1. A message notification device, comprising: The line-of-sight detection unit detects the direction of the driver's line of sight. The notification unit, when controlling the vehicle in an autonomous driving mode that does not require driver input and when the driver's gaze is not directed in a direction related to the driving operation, further detects, within a predetermined time after detecting changes in the vehicle's surrounding environment to the extent that the driver's attention is directed forward, changes in the vehicle's behavior to the extent that the driver can perceive, or operations by the driver on the in-vehicle equipment mounted in front of the driver's seat, that the driver's gaze is directed in a direction related to the driving operation, displays a message that the driver should be aware of on a display unit located in front of the driver's seat; and The determination unit independently determines the message that the driver should be informed of, based on the degree to which the driver focuses their attention forward, changes in the vehicle's surrounding environment, changes in the vehicle's behavior that the driver can perceive, and the detection of the driver's operation of the on-board equipment.

2. The message notification device according to claim 1, The determining unit determines the information that the driver should know based on changes in the environment inside the vehicle or changes in the surrounding environment of the vehicle to the extent that the driver is not paying attention to what is ahead.

3. The message notification device according to claim 1, The changes in the surrounding environment include at least one of the following: changes in illumination around the vehicle, changes in air pressure around the vehicle, and changes in the interval from the vehicle to a preceding vehicle traveling in front of the vehicle.

4. The message notification device according to claim 1, The changes in vehicle behavior include at least one of the following: starting or stopping the operation of the windshield wipers, turning the headlights on or off, and stopping or slowing down the vehicle.

5. The message notification device according to claim 1, The operation of the in-vehicle equipment includes the operation of at least one of the vehicle navigation device, audio equipment, and air conditioning equipment.

6. A message notification method, comprising: When the vehicle is controlled in an autonomous driving mode that does not require driver operation, and the driver's gaze is not directed in a direction related to the driving operation, if within a predetermined time after detecting changes in the vehicle's surrounding environment to the extent that the driver's attention is directed forward, detecting changes in the vehicle's behavior to the extent that the driver can perceive, or detecting the driver's operation of the in-vehicle equipment mounted in front of the driver's seat, a message that the driver should be aware of is displayed on the display unit located in front of the driver's seat. The information that the driver should be informed of is determined independently of the degree to which the driver focuses their attention forward, changes in the vehicle's surrounding environment, changes in the vehicle's behavior that the driver can perceive, and the detection of the driver's operation of the on-board equipment.

7. The message notification method according to claim 6 further includes: Based on changes in the vehicle's interior environment or changes in the vehicle's surrounding environment to the extent that the driver is not paying attention to what is ahead, determine the information that the driver should be informed of.

8. The message notification method according to claim 6, In the display, if, within a predetermined time after detecting a change in the surrounding environment, a change in the vehicle's behavior, or an operation by the driver, the driver's gaze is further detected to be directed in a direction related to the driving operation, the message is displayed on the display unit.

9. The message notification method according to claim 6, The changes in the surrounding environment include at least one of the following: changes in illumination around the vehicle, changes in air pressure around the vehicle, and changes in the interval from the vehicle to a preceding vehicle traveling in front of the vehicle.

10. The message notification method according to claim 6, The changes in vehicle behavior include at least one of the following: starting or stopping the operation of the windshield wipers, turning the headlights on or off, and stopping or slowing down the vehicle.

11. The message notification method according to claim 6, The operation of the in-vehicle equipment includes the operation of at least one of the vehicle navigation device, audio equipment, and air conditioning equipment.

12. A computer program for message notification, which causes a computer mounted in a vehicle to execute: When the vehicle is controlled in an autonomous driving mode that does not require driver operation, and the driver's gaze is not directed in a direction related to the driving operation, if within a predetermined time after detecting changes in the vehicle's surrounding environment to the extent that the driver's attention is directed forward, detecting changes in the vehicle's behavior to the extent that the driver can perceive, or detecting the driver's operation of the in-vehicle equipment mounted in front of the driver's seat, a message that the driver should be aware of is displayed on the display unit located in front of the driver's seat. The information that the driver should be informed of is determined independently of the degree to which the driver focuses their attention forward, changes in the vehicle's surrounding environment, changes in the vehicle's behavior that the driver can perceive, and the detection of the driver's operation of the on-board equipment.

13. The computer program for message notification according to claim 12, further comprising the computer performing: Based on changes in the vehicle's interior environment or changes in the vehicle's surrounding environment to the extent that the driver is not paying attention to what is ahead, determine the information that the driver should be informed of.

14. The computer program for message notification according to claim 12, In the display, if, within a predetermined time after detecting a change in the surrounding environment, a change in the vehicle's behavior, or an operation by the driver, the driver's gaze is further detected to be directed in a direction related to the driving operation, the display unit will display the message.

15. The computer program for message notification according to claim 12, The changes in the surrounding environment include at least one of the following: changes in illumination around the vehicle, changes in air pressure around the vehicle, and changes in the interval from the vehicle to a preceding vehicle traveling in front of the vehicle.

16. The computer program for message notification according to claim 12, The changes in vehicle behavior include at least one of the following: starting or stopping the operation of the windshield wipers, turning the headlights on or off, and stopping or slowing down the vehicle.

17. The computer program for message notification according to claim 12, The operation of the in-vehicle equipment includes the operation of at least one of the vehicle navigation device, audio equipment, and air conditioning equipment.

Images