Notification device, notification method, and computer program for notification

Abstract

To notify a driver of a vehicle of messages at an appropriate time according to the driver's condition.SOLUTION: A notification device determines the presence or absence of a face of a driver of a vehicle in a first area of a first image by infrared light, and determines the presence or absence of an object different from the driver's face in a second area corresponding to the first area in a second image by visible light. When determining that there is no object in the second area, the notification device sets a first time as a grace time, and when there is an object in the second area, the notification device sets a second time longer than the first time as a grace time. The notification device outputs a notification to the driver via an output device after the grace time has elapsed after the face of the driver is determined to be not in the first area.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a notification device, a notification method, and a notification computer program that output a notification to a driver according to the state of the driver of a vehicle.

Background Art

[0002] The driver monitoring device described in Patent Document 1 can accurately set the orientation or position of the face of a reference driver from each face image obtained by a driver monitoring camera that photographs the driver.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Even when facial parts such as eyes, nose, and mouth cannot be detected at normal positions from a face image, depending on the cause, a notification to the driver may not be necessary. For example, if the driver's face is out of the camera's field of view due to a slouch caused by poor physical condition, it is difficult to continue appropriate driving operations, so it is preferable to seek a response to the situation by notification. On the other hand, if only a part of the body such as the driver's hand or arm, or an object held by the driver such as a smartphone or food and drink, enters between the face and the camera as an obstacle and temporarily makes the face undetectable, no special notification is necessary. If it is uniformly notified that the driver's face cannot be detected from the face image, a notification that is not originally necessary will be made.

[0005] An object of the present disclosure is to provide a notification device that can notify a driver of a vehicle with a message at an appropriate timing according to the state of the driver.

Means for Solving the Problems

[0006] The gist of the present disclosure is as follows.

[0007] (1) A face determination unit that determines the presence or absence of the face of the driver of the vehicle in a first region of a first image by infrared light, An object determination unit that determines the presence or absence of an object different from the face of the driver in a second region corresponding to the first region in a second image by visible light, A notification unit that outputs a notification to the driver via an output device after a predetermined grace period has elapsed since it is determined that there is no face of the driver in the first region, A setting unit that sets a first time as the grace period when it is determined that there is no object in the second region, and sets a second time longer than the first time as the grace period when it is determined that there is an object in the second region, A notification device comprising:

[0008] (2) When it is determined that there is an object in the second region, the object determination unit further determines the object type into which the object is classified, The second time is set in advance for each object type, and the notification device according to (1) above.

[0009] (3) Further comprising a steering determination unit that determines whether or not the driver is holding a steering member for steering the vehicle, When it is determined that there is an object in the second region, the setting unit sets a third time longer than the second time as the grace period when it is determined that the driver is holding the steering member, and the notification device according to (1) or (2) above.

[0010] (4) A notification device that outputs a notification to the driver according to the state of the driver of the vehicle, determines the presence or absence of the face of the driver in a first region of a first image by infrared light, determines the presence or absence of an object different from the face of the driver in a second region corresponding to the first region in a second image by visible light, When it is determined that the object is not in the second area, a first time is set as the grace period, and when it is determined that the object is in the second area, a second time longer than the first time is set as the grace period. After the grace period has elapsed since it is determined that the driver's face is not in the first area, a notification to the driver is output via an output device. A notification method including this.

[0011] (5) Determine the presence or absence of the face of the driver of the vehicle in the first area of the first image by infrared light. Determine the presence or absence of an object different from the face of the driver in the second area corresponding to the first area in the second image by visible light. When it is determined that the object is not in the second area, a first time is set as the grace period, and when it is determined that the object is in the second area, a second time longer than the first time is set as the grace period. After the grace period has elapsed since it is determined that the driver's face is not in the first area, a notification to the driver is output via an output device. A notification computer program for causing a computer to execute a process including this.

[0012] According to the notification device according to the present disclosure, a message can be notified to the driver of the vehicle at an appropriate timing according to the state of the driver.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Best Mode for Carrying Out the Invention

[0014] Hereinafter, with reference to the drawings, a notification device that can notify a vehicle driver of a message at an appropriate timing according to the driver's state will be described in detail. The notification device determines the presence or absence of the face of the vehicle driver in a first area of a first image using infrared light. Further, the notification device determines the presence or absence of an object different from the driver's face in a second area corresponding to the first area in a second image using visible light. When it is determined that there is no object in the second area, the notification device sets a first time as a grace period, and when it is determined that there is the object in the second area, the notification device sets a second time longer than the first time as the grace period. Then, after the grace period has elapsed since it is determined that there is no driver's face in the first area, the notification device outputs a notification to the driver via an output device.

[0015] FIG. 1 is a schematic configuration diagram of a vehicle in which the notification device is mounted.

[0016] The vehicle 1 includes a surrounding camera 2, an infrared light camera 3, a visible light camera 4, a speaker device 5, a GNSS (Global Navigation Satellite System) receiver 6, a storage device 7, a travel control device 8, and a notification device 9. The surrounding camera 2, the infrared light camera 3, the visible light camera 4, the speaker device 5, the GNSS receiver 6, and the storage device 7, and the travel control device 8 and the notification device 9 are communicably connected via an in-vehicle network conforming to a standard such as a controller area network.

[0017] The surrounding camera 2 is an example of a surrounding sensor for generating surrounding data representing the situation around the vehicle 1. The surrounding camera 2 includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The surrounding camera 2 is arranged, for example, at the upper front inside the vehicle cabin, facing forward, and photographs the situation around the vehicle 1 through the windshield at a predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second), and outputs a surrounding image as surrounding data representing the situation around the vehicle 1.

[0018] The infrared light camera 3 is an example of a driver sensor for photographing the face of the driver of the vehicle and generating a first image using infrared light. The infrared light camera 3 includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. In addition, the infrared light camera 3 has a light source that emits infrared light. The infrared light camera 3 is attached, for example, in front of the vehicle cabin, facing the face of the driver sitting on the driver's seat. The infrared light camera 3 irradiates the driver with infrared light at a predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second) and outputs the first image in time series.

[0019] The visible light camera 4 is an example of a driver sensor for photographing the face of the driver of the vehicle and generating a second image using visible light. The visible light camera 4 includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector. The visible light camera 4 is attached, for example, in front of the vehicle cabin, facing the face of the driver sitting on the driver's seat. The visible light camera 4 executes photographing at a predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second) and outputs the second image in time series. The infrared light camera 3 and the visible light camera 4 may be configured to share a part of their respective optical systems.

[0020] The speaker device 5 is an example of an output device and has, for example, an amplifier and a speaker unit. The speaker device 5 outputs voice representing a message according to a signal received from the notification device 9 via the in-vehicle network. The vehicle 1 may have a display device having a liquid crystal display as an output device. Further, the output device may be a light source that lights up in a pattern according to the content of the message, a vibrator that vibrates in a pattern according to the content of the message, or the like.

[0021] The GNSS receiver 6 is an example of a positioning sensor, receives GNSS signals from GNSS satellites at a predetermined cycle, and measures the self-position of the vehicle 1 based on the received GNSS signals. The GNSS receiver 6 outputs a positioning signal representing the measurement result of the self-position of the vehicle 1 based on the GNSS signals to the travel control device 8 via the in-vehicle network at a predetermined cycle.

[0022] The storage device 7 is an example of a storage unit and has, for example, a hard disk device or a non-volatile semiconductor memory. The storage device 7 stores a high-precision map. The high-precision map includes, for example, information representing lane dividing lines that divide lanes in a predetermined area represented by the high-precision map.

[0023] The travel control device 8 is an ECU (Electronic Control Unit) including a communication interface, a memory, and a processor. The travel control device 8 reads information on lane demarcation lines around the vehicle position represented by the positioning signal received from the GNSS receiver 6 from the storage device 7 that stores the high-precision map. The travel control device 8 detects the surrounding lane demarcation lines from the surrounding image received from the surrounding camera 2, and identifies the lane in which the vehicle 1 is traveling by performing matching with the lane demarcation lines in the high-precision map. The travel control device 8 detects surrounding objects such as other vehicles around from the surrounding image received from the surrounding camera 2, maintains a predetermined distance from the object, and creates a travel route so as to maintain or change the lane according to the situation. Then, the travel control device 8 outputs a control signal to the travel mechanism (not shown) of the vehicle 1 so as to travel along the travel route. The surrounding camera 2, GNSS receiver 6, storage device 7, and travel control device 8 that cooperate to realize autonomous driving are an example of an autonomous driving system. The travel mechanism includes, for example, an engine or a motor that supplies power to the vehicle 1, an accelerator for adjusting the acceleration of the vehicle 1, a brake for reducing the travel speed of the vehicle 1, and a steering mechanism for steering the vehicle 1.

[0024] The travel control device 8 can control the travel of the vehicle 1 according to an autonomous driving level (for example, autonomous driving level 2 defined by the Society of Automotive Engineers (SAE)) that controls the travel of the vehicle 1 under the supervision of the driver. The notification device 9 detects a state in which it is considered that the driver cannot take over the driving operation, such as when the surrounding of the vehicle 1 is not properly monitored, and outputs a notification. If the driver's state is not improved even by such a notification, the travel control device 8 terminates the control according to the above-described autonomous driving level 2 and executes control to ensure safety such as stopping on the shoulder.

[0025] The notification device 9 is an ECU including a communication interface, a memory, and a processor. The notification device 9 notifies the driver of a message via the speaker device 5 based on the state of the driver detected from each of the first image received from the infrared light camera 3 and the second image received from the visible light camera 4. The notification device 9 may be implemented on the same ECU as the ECU on which the travel control device 8 is implemented.

[0026] Figure 2 is a hardware schematic diagram of the notification device 9. The notification device 9 includes a communication interface 91, a memory 92, and a processor 93.

[0027] The communication interface 91 is an example of a communication unit and has a communication interface circuit for connecting the notification device 9 to the in-vehicle network. The communication interface 91 supplies the received data to the processor 93. Also, the communication interface 91 outputs the data supplied from the processor 93 to the outside.

[0028] The memory 92 is an example of a storage unit and has a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 92 stores various data used for processing by the processor 93. For example, the memory 92 stores first region position information representing the position of a first region, which is a standard region where the driver's face is located in the first image, and second region position information representing the position of a second region corresponding to the first region in the second image. Also, the memory 92 stores the parameters of a neural network that operates as an identifier for detecting face feature points from the first region and the parameters of a neural network that operates as an identifier for detecting an object from the second region. Also, the memory 92 stores a first time and a second time set as a grace period, voice information to be notified to the driver, etc. Also, the memory 92 stores various application programs, for example, a notification computer program for executing notification processing.

[0029] The processor 93 is an example of a control unit and includes one or more processors and their peripheral circuits. The processor 93 may further include other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit.

[0030] FIG. 3 is a functional block diagram of the processor 93 included in the notification device 9.

[0031] The processor 93 of the notification device 9 includes, as functional blocks, a face determination unit 931, an object determination unit 932, a notification unit 933, and a setting unit 934. Each of these units included in the processor 93 is a functional module implemented by a computer program stored in the memory 92 and executed on the processor 93. The computer program for realizing the functions of each unit of the processor 93 may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. Alternatively, each of these units included in the processor 93 may be implemented in the notification device 9 as an independent integrated circuit, a microprocessor, or firmware.

[0032] The face determination unit 931 detects the positions of face feature points from the first image by inputting the first image acquired from the infrared camera 3 into a discriminator that has been pre-trained to detect face feature points such as the inner corners of the eyes, the outer corners of the eyes, and the corners of the mouth. Note that the face determination unit 931 may input only the first region of the first image into the discriminator. The discriminator may be a convolutional neural network (CNN) that has been trained to detect face feature points using a large number of infrared images including face feature points as teacher data in advance.

[0033] If no face feature points are detected in the first region pre-stored in the memory 92, the face determination unit 931 determines that there is no driver's face in the first region.

[0034] Further, the face determination unit 931 may identify a standard three-dimensional model of a face corresponding to the face feature points of the driver detected in the first region, and determine the presence or absence of the driver's face in the first region based on the relationship between the face region and the first region in the three-dimensional model according to a predetermined condition.

[0035] For example, the face determination unit 931 identifies a three-dimensional model in which the deviation between the detected other face feature points and the corresponding face feature points of the three-dimensional model is minimized when the position of a certain face feature point among the detected one or more face feature points is adjusted to the position of the corresponding face feature point of the three-dimensional model. When a predetermined condition such as 80% or more of the face region in the three-dimensional model is within the first region, or 50% or more of the first region is the face region is satisfied, the face determination unit 931 determines that the driver's face is present in the first region.

[0036] FIG. 4(a) is a first example of the first image, and FIG. 4(b) is a second example of the first image. The infrared camera 3 generates an image P1a as the first image at a certain time ta. Further, the infrared camera 3 generates an image P1b as the first image at a time tb different from the time ta.

[0037] The face determination unit 931 determines that the driver's face is present in the first region R1 of the image P1a based on the face feature points of the driver's face F detected from the first region R1 set in the image P1a. On the other hand, in the image P1b, an object OBJ (for example, the driver's hand) is reflected in the first region R1, and the face feature points of the driver's face F are not appropriately detected. Therefore, the face determination unit 931 determines that the driver's face is not present in the first region R1 of the image P1b.

[0038] Returning to FIG. 3, the object determination unit 932 inputs the second image acquired from the visible light camera 4 into an identifier that has been pre-learned to detect an object, and detects an object (for example, a smartphone, a plastic bottle, a hand, an arm, etc.) from the second image. The identifier may be a CNN that has been learned to detect an object using a large number of visible light images in which the object is represented as teacher data in advance.

[0039] When an object is detected in the second image by the object determination unit 932 and the relationship between the area represented by the object and the second area satisfies a predetermined condition (for example, 30% or more of the area represented by the object is within the second area, or 10% or more of the second area is within the area represented by the object, etc.), it is determined that there is an object in the second area.

[0040] For example, when an image such as the image P1b shown in FIG. 4(b) is generated as the second image by the visible light camera 4, the object determination unit 932 determines the presence or absence of an object in the second area based on the relationship between the area represented by the object OBJ and the second area set in the second image.

[0041] Since the visible light camera 4 generates an image based on ambient light, it may not be possible to appropriately determine the presence or absence of the driver's face from the second image generated when the ambient light amount is insufficient, such as at night. On the other hand, since the first image by the infrared light camera 3 does not include color information, it is not easy to appropriately detect objects having various types from the first image. By determining the presence or absence of a face based on the first image by the infrared light camera 3 and detecting the presence or absence of an object based on the second image by the visible light camera 4, it becomes possible to notify the driver of the vehicle at an appropriate timing according to the driver's state.

[0042] After it is determined that there is no driver's face in the first area, the notification unit 933 outputs a notification to the driver via the speaker device 5 after the elapse of a grace period set in the setting unit 934 described later. The notification to the driver is, for example, an audio such as "Your posture is out of alignment. If your posture does not return, stop on the road shoulder." Further, the notification unit 933 may notify the driver by operating the light source, vibrator, etc. that the vehicle 1 has as output devices so as to correspond to the message to be notified to the driver.

[0043] When it is determined that there is no object in the second area, the setting unit 934 sets a first time (for example, 0.5 seconds) as the grace period, and when it is determined that there is an object in the second area, sets a second time (for example, 3 seconds) longer than the first time as the grace period.

[0044] The setting unit 934 may be configured not to change the grace period that has already started and has been determined to have no face in the first area midway. For example, at time ti, it is determined by the face determination unit 931 that there is no face in the first area and by the object determination unit 932 that there is an object in the second area, and it is assumed that the grace period with the second time set has started. Then, at time tj when the second time has not elapsed since time ti, it is again determined by the face determination unit 931 that there is no face in the first area, and this time it is determined by the object determination unit 932 that there is no object in the second area. In this case, the notification may be after the elapse of the second time from time ti instead of after the elapse of the first time from time tj.

[0045] FIG. 5 is a flowchart of the notification process. The notification device 9 repeatedly executes the process shown in FIG. 5 at a predetermined time interval (e.g., 1 second interval) while the vehicle 1 is running under the control of the driving control device 8.

[0046] First, the face determination unit 931 of the processor 93 of the notification device 9 determines the presence or absence of the driver's face in the first area of the first image generated by the infrared light camera 3 (step S1). If it is determined that there is no face (step S1: N), the notification process ends.

[0047] If it is determined that there is the driver's face in the first area (step S1: Y), the object determination unit 932 of the processor 93 determines the presence or absence of an object in the second area of the second image generated by the visible light camera 4 (step S2).

[0048] If it is determined that there is no object in the second area (step S2: N), the setting unit 934 of the processor 93 sets the first time as the grace period (step S3). On the other hand, if it is determined that there is an object in the second area (step S2: Y), the setting unit 934 sets the second time, which is longer than the first time, as the grace period (step S4).

[0049] Subsequently, after the grace period has elapsed since it is determined that the driver's face is not in the first region, the notification unit 933 of the processor 93 outputs a notification to the driver via the speaker device 5 (step S5) and ends the notification process.

[0050] By executing the notification process in this way, the notification device 9 can notify the driver of the vehicle at an appropriate timing according to the driver's state.

[0051] In the flowchart of the above-described notification process, the determination of the presence or absence of a face in the first region (step S1) and the determination of the presence or absence of an object in the second region (step S2) may be executed in the reverse order (step S2 first). In this case, step S1 may be executed before or after the setting of the grace period (steps S3, S4) by the setting unit 934 as long as it is between step S2 and the output of the notification by the notification unit 933 (step S5).

[0052] In a modification, the vehicle 1 may be provided with a steering holding sensor (not shown) on a steering wheel (not shown), which is an example of a steering member for steering the vehicle 1. In this case, the processor 93 of the notification device 9 has a steering holding determination unit that determines whether the driver is holding the steering member according to a steering holding signal output from the steering holding sensor. When it is further determined that the driver is holding the steering member when it is determined that there is an object in the second region, it may be assumed that the driver can perform appropriate driving operations. At this time, the setting unit 934 may set a third time (for example, 5 seconds), which is longer than the second time, as the grace period.

[0053] In such a modification, a message can be notified to the driver of the vehicle 1 at an appropriate timing according to the steering holding state of the driver of the vehicle 1.

[0054] In another modification, the object determination unit 932 may further determine the object type into which the object determined to be in the second region is classified. In this case, the setting unit 934 may set a second time to be set as a grace period when it is determined that there is an object in the second region, according to the object type of the object.

[0055] In such a modification, a message can be notified to the driver of the vehicle 1 at an appropriate timing according to the type of the object existing in the face region of the driver of the vehicle 1.

[0056] Those skilled in the art should understand that various changes, substitutions, and modifications can be made to this without departing from the spirit and scope of the present disclosure.

Description of Reference Numerals

[0057] 1 Vehicle 3 Infrared Camera 4 Visible Light Camera 5 Speaker Device 9 Notification Device 931 Face Determination Unit 932 Object Determination Unit 933 Notification Unit 934 Setting Unit

Claims

1. A face determination unit that determines the presence or absence of the face of a driver of a vehicle in a first region of a first image by infrared light, An object determination unit that determines the presence or absence of an object different from the face of the driver in a second region corresponding to the first region in a second image by visible light, A notification unit that outputs a notification to the driver via an output device after a predetermined grace period has elapsed since it is determined that there is no face of the driver in the first region, A setting unit that sets a first time as the grace period when it is determined that there is no object in the second region, and sets a second time longer than the first time as the grace period when it is determined that there is an object in the second region, A notification device comprising the above.

2. When the object determination unit determines that there is an object in the second region, the object determination unit further determines the object type to which the object belongs, The second time is set in advance for each object type, and the notification device according to claim 1.

3. The notification device further comprises a steering determination unit that determines whether the driver is holding a steering member for steering the vehicle, When it is determined that there is an object in the second region, the setting unit sets a third time longer than the second time as the grace period when it is determined that the driver is holding the steering member, and the notification device according to claim 1 or 2.

4. A notification device that outputs a notification to the driver according to the state of the driver of the vehicle, Determines the presence or absence of the face of the driver in a first region of a first image by infrared light, Determines the presence or absence of an object different from the face of the driver in a second region corresponding to the first region in a second image by visible light, When it is determined that there is no object in the second region, sets a first time as the grace period, and when it is determined that there is an object in the second region, sets a second time longer than the first time as the grace period, After the grace period has elapsed since it is determined that there is no face of the driver in the first region, outputs a notification to the driver via an output device, A notification method including this.

5. Determines the presence or absence of the face of the driver of the vehicle in a first region of a first image by infrared light, Determines the presence or absence of an object different from the face of the driver in a second region corresponding to the first region in a second image by visible light, When it is determined that the object is not in the second area, a first time is set as a grace period. When it is determined that the object is in the second area, a second time longer than the first time is set as the grace period. After the grace period has elapsed since it is determined that the driver's face is not in the first area, a notification to the driver is output via an output device. A notification computer program for causing a computer to execute a process including this.

Images