Driver monitoring device, driver monitoring method, and computer program for driver monitoring

Abstract

To provide a driver monitoring device capable of appropriately detecting an abnormality occurring in a driver.SOLUTION: A driver monitoring device includes: a posture detection section 31 that detects a posture of a driver on the basis of at least any of a plurality of images in a cabin of a vehicle 10 generated in time series by an imaging section 2 provided in the vehicle 10; a heart rate detection section 32 that detects a heart rate of the driver on the basis of the plurality of images; an abnormality posture determination section 33 that determines whether or not the detected posture of the driver is a predetermined abnormality posture; a heart rate abnormality determination section 34 that determines whether or not there is an abnormality of the heart rate of the driver on the basis of the detected heart rate when the detected posture of the driver is the predetermined abnormality posture; and a notification processing section 35 that notifies the driver of an alert via a notification device 3 provided in the cabin when it is determined that there is the abnormality of the heart rate.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a driver monitoring device, a driver monitoring method, and a computer program for driver monitoring that monitor a driver of a vehicle.

Background Art

[0002] When level 3 automated driving as defined by the Society of Automotive Engineers (SAE) is applied to a vehicle, the driver is allowed to perform non-driving actions such as operating a smartphone (hereinafter sometimes referred to as non-driving actions), on the condition that the driver maintains a state where they can take over the driving of the vehicle. However, for some reason, the driver may be required to take over the driving of the vehicle. Therefore, the state of the driver is monitored. Thus, a technique for monitoring the state of the driver based on an image of the driver has been proposed (see Patent Document 1).

[0003] In the technique disclosed in Patent Document 1, the driver monitoring device detects a slouching of the driver based on an image of the driver's seat in the vehicle, and determines whether the detected slouching of the driver is due to the driver's habit. Then, this driver monitoring device notifies the driver about the slouching in different manners depending on whether it is determined to be slouching due to a habit or slouching other than due to a habit. This driver monitoring device determines that it is slouching due to a habit when the slouching of the driver has been continuously detected for a period longer than the slouching determination time since immediately after the start of driving.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Depending on the driver, it may be difficult to distinguish between the posture during non-driving behavior and the posture during abnormalities such as diseases that occur to the driver. In such a case, the posture during non-driving behavior may be erroneously detected as the posture during abnormalities, and as a result, the driver may be given an unnecessary warning.

[0006] Therefore, an object of the present invention is to provide a driver monitoring device that can appropriately detect abnormalities that occur to the driver.

Means for Solving the Problems

[0007] According to one embodiment, a driver monitoring device is provided. This driver monitoring device includes a posture detection unit that detects the posture of the driver of the vehicle based on at least any one of a plurality of images in the vehicle interior of the vehicle generated in time series by an imaging unit provided in the vehicle, a heart rate detection unit that detects the heart rate of the driver based on those plurality of images, an abnormal posture determination unit that determines whether the detected posture of the driver is a predetermined abnormal posture, a heart rate abnormality determination unit that determines the presence or absence of an abnormality in the driver's heart rate based on the detected heart rate when the detected posture of the driver is a predetermined abnormal posture, and a notification processing unit that notifies the driver of a warning via a notification device provided in the vehicle interior when it is determined that there is an abnormality in the heart rate.

[0008] In this driver monitoring device, the abnormal posture determination unit further determines whether the detected posture of the driver corresponds to either a first abnormal posture or a second abnormal posture among the predetermined abnormal postures, the heart rate abnormality determination unit determines the presence or absence of an abnormality in the driver's heart rate only when the detected posture of the driver is the first abnormal posture, and the notification processing unit preferably notifies the driver of a warning via the notification device regardless of the presence or absence of an abnormality in the heart rate when the detected posture of the driver is the second abnormal posture.

[0009] Also, in this driver monitoring device, the heart rate detection unit preferably detects the heart rate of the driver only when it is determined that the detected posture of the driver is a predetermined abnormal posture.

[0010] According to another embodiment, a driver monitoring method is provided. This driver monitoring method detects the posture of the driver of a vehicle based on at least any one of a plurality of images of the interior of the vehicle generated in time series by an imaging unit provided in the vehicle, detects the heart rate of the driver based on the plurality of images, determines whether the detected posture of the driver is a predetermined abnormal posture, and if the detected posture of the driver is a predetermined abnormal posture, determines the presence or absence of an abnormality in the driver's heartbeat based on the detected heart rate, and notifies the driver of a warning via a notification device provided in the vehicle interior when it is determined that there is an abnormality in the heartbeat.

[0011] According to still another embodiment, a computer program for driver monitoring is provided. This computer program for driver monitoring detects the posture of the driver of a vehicle based on at least any one of a plurality of images of the interior of the vehicle generated in time series by an imaging unit provided in the vehicle, detects the heart rate of the driver based on the plurality of images, determines whether the detected posture of the driver is a predetermined abnormal posture, and if the detected posture of the driver is a predetermined abnormal posture, determines the presence or absence of an abnormality in the driver's heartbeat based on the detected heart rate, and notifies the driver of a warning via a notification device provided in the vehicle interior when it is determined that there is an abnormality in the heartbeat, and includes instructions for causing a processor mounted on the vehicle to execute the above.

Advantages of the Invention

[0012] The driver monitoring device according to the present disclosure has an effect of being able to appropriately detect an abnormality occurring in the driver.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0014] Hereinafter, with reference to the drawings, a driver monitoring device, a driver monitoring method and a computer program for driver monitoring executed by the driver monitoring device will be described. When the posture of the driver detected from the image in the vehicle interior is a predetermined abnormal posture, this driver monitoring device determines the presence or absence of an abnormality in the driver's heartbeat. And when it is determined that there is an abnormality in the driver's heartbeat, this driver monitoring device notifies the driver of a warning via a notification device provided in the vehicle interior. Thereby, this driver monitoring device suppresses misdetection of an abnormality occurring in the driver when the driver is not taking a posture suitable for driving (hereinafter sometimes simply referred to as a driving posture), such as when the driver is performing a non-driving action even though no abnormality has occurred.

[0015] FIG. 1 is a schematic configuration diagram of a vehicle control system including a driver monitoring device. In the present embodiment, a vehicle control system 1 mounted on a vehicle 10 and controlling the vehicle 10 includes a driver monitoring camera 2, a notification device 3, and an electronic control unit (ECU) 4 which is an example of a driver monitoring device. The driver monitoring camera 2, the notification device 3, and the ECU 4 are communicably connected to each other via an in-vehicle network compliant with a communication standard such as a controller area network. Note that the vehicle control system 1 may have an outside camera (not shown) that captures an area around the vehicle 10 and generates an image representing the area around the vehicle. Alternatively, the vehicle control system 1 may have a distance sensor (not shown) such as a LiDAR or a radar that measures the distance from the vehicle 10 to an object existing around the vehicle 10. Further, the vehicle control system 1 may have a positioning device (not shown) such as a GPS receiver for positioning the position of the vehicle 10 based on a signal from a satellite. Furthermore, the vehicle control system 1 may have a navigation device (not shown) for searching for a planned driving route to a destination. Furthermore, the vehicle control system 1 may have a storage device (not shown) that stores map information referred to in the automatic driving control of the vehicle 10.

[0016] The vehicle control system 1 can control the vehicle 10 at a driving control level of level 3 according to the definition of SAE, that is, a driving control level that does not require the driver to operate the accelerator, brake, and steering and monitor the surroundings of the vehicle. And while the ECU 4 controls the vehicle 10 at the driving control level of level 3, it executes driver monitoring processing. However, not limited to this, the ECU 4 may also execute driver monitoring processing even while the vehicle 10 is being controlled at a driving control level at which the driver is involved in the driving of the vehicle 10, for example, at a driving control level of levels 0 to 2 according to the definition of SAE.

[0017] The driver monitoring camera 2 is an example of an imaging unit, and includes a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light and near-infrared light, such as a CCD or a C-MOS, and an imaging optical system that forms an image of an area to be photographed on the two-dimensional detector. The driver monitoring camera 2 may further include a light source for illuminating the driver, such as an LED that emits near-infrared light. The driver monitoring camera 2 is attached, for example, to the instrument panel or in its vicinity facing the driver so that the head of the driver sitting on the driver's seat of the vehicle 10 is included in the photographed area, that is, so that the head of the driver can be photographed. The driver monitoring camera 2 photographs the driver at a predetermined photographing cycle (for example, 1 / 30 second to 1 / 10 second) and generates an image in which the driver is shown (hereinafter referred to as a driver image). In the present embodiment, the driver image is generated as an image in which each pixel represents the luminance of near-infrared light. Each time the driver monitoring camera 2 generates a driver image, it outputs the generated driver image to the ECU 4 via the in-vehicle network.

[0018] The notification device 3 is provided in the passenger compartment of the vehicle 10 and is a device that gives a predetermined notification to the driver by light, voice, vibration, character display, or image display. For this purpose, the notification device 3 has, for example, at least one of a speaker, a light source, a vibrator, or a display device. When the notification device 3 receives a notification representing a warning from the ECU 4 to the driver, it notifies the driver of the warning by voice from the speaker, emission or blinking of the light source, vibration of the vibrator, or display of a warning message on the display device.

[0019] The ECU 4 controls the operation of the vehicle 10 according to the operation control level applied to the vehicle 10. Further, the ECU 4 monitors the driver based on the driver image received from the driver monitoring camera 2 and detects an abnormality occurring to the driver. When the ECU 4 determines that an abnormality has occurred to the driver, it controls the vehicle 10 to warn the driver or to stop the vehicle 10 urgently.

[0020] Figure 2 is a hardware configuration diagram of the ECU 4. As shown in Figure 2, the ECU 4 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may each be configured as separate circuits, or may be integrally configured as a single integrated circuit.

[0021] The communication interface 21 has an interface circuit for connecting the ECU 4 to the in-vehicle network. And each time the communication interface 21 receives a driver image from the driver monitoring camera 2, it passes the received driver image to the processor 23. Furthermore, when the communication interface 21 receives a signal including information to be notified to the driver via the notification device 3, such as a notification representing a warning to the driver from the processor 23, it outputs the signal to the notification device 3.

[0022] The memory 22 is an example of a storage unit and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. And the memory 22 stores various algorithms and various data used in the driver monitoring process executed by the processor 23 of the ECU 4. For example, the memory 22 stores various parameters used for detecting the driver's posture. Furthermore, the memory 22 stores parameters for defining determination conditions (hereinafter sometimes simply referred to as abnormal determination conditions) for determining whether the driver's posture is a predetermined abnormal posture. Furthermore, the memory 22 temporarily stores the driver image and various data generated during the driver monitoring process. Furthermore, the memory 22 stores various parameters and various data used for controlling the operation of the vehicle 10. Such data includes images generated by an external camera, ranging signals generated by a distance sensor, a positioning signal representing the position of the vehicle 10 generated by a GPS receiver, a planned driving route generated by a navigation device, and map information.

[0023] The processor 23 has one or more CPUs (Central Processing Units) and its peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logical arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. And the processor 23 executes driving control processing of the vehicle including driver monitoring processing at a predetermined cycle.

[0024] FIG. 3 is a functional block diagram of the processor 23 regarding driving control processing of the vehicle including driver monitoring processing. The processor 23 has a posture detection unit 31, a heartbeat detection unit 32, an abnormal posture determination unit 33, a heartbeat abnormality determination unit 34, a notification processing unit 35, and a vehicle control unit 36. Each of these units included in the processor 23 is a functional module realized by, for example, a computer program operating on the processor 23. Alternatively, each of these units included in the processor 23 may be a dedicated arithmetic circuit provided in the processor 23. Among these units included in the processor 23, the processing executed by the posture detection unit 31, the heartbeat detection unit 32, the abnormal posture determination unit 33, the heartbeat abnormality determination unit 34, and the notification processing unit 35 corresponds to the driver monitoring processing.

[0025] The posture detection unit 31 detects the driver's posture from at least any one of a plurality of time-series driver images received by the ECU 4 from the driver monitoring camera 2, for example, the latest driver image. In the present embodiment, the posture detection unit 31 detects the position of the driver's face and the direction of the driver's face as information representing the driver's posture.

[0026] The posture detection unit 31 detects, for example, a region in the driver image where the driver's face appears (hereinafter referred to as the face region) by inputting the driver image into a discriminator that has been pre-trained to detect the driver's face from the image. As such a discriminator, the posture detection unit 31 can use, for example, a so-called deep neural network (DNN) or AdaBoost discriminator having a convolutional neural network (CNN) type architecture. Further, the posture detection unit 31 detects a plurality of feature points of the driver's face, such as the outer corner of the eye, the inner corner of the eye, the tip of the nose, and the corners of the mouth, from the face region of the driver image. At this time, the posture detection unit 31 detects the feature points of the face by inputting the face region into a discriminator that has been pre-trained to detect the feature points of the face represented in the image. As such a discriminator, the posture detection unit 31 can use, for example, a DNN having a CNN type architecture, a support vector machine, or an AdaBoost discriminator. Note that the discriminator for face region detection and the discriminator for face feature point detection may be integrally configured. In this case, the posture detection unit 31 can detect each of the face region and the individual feature points of the face by inputting the driver image into the discriminator. Alternatively, the posture detection unit 31 may detect the individual feature points of the driver's face from the face region according to template matching between a template representing the feature points of the face and the face region, or other methods for detecting the feature points of the face.

[0027] The posture detection unit 31 fits the detected individual feature points of the face to a three-dimensional face model representing the three-dimensional shape of the face. Then, the posture detection unit 31 detects the orientation of the face of the three-dimensional face model when each feature point fits best to the three-dimensional face model as the orientation of the driver's face. Note that the posture detection unit 31 may detect the orientation of the driver's face based on the driver image according to other methods for determining the orientation of the face represented in the image.

[0028] Furthermore, the posture detection unit 31 detects the position of the driver's face based on the center of gravity position of the face region in the driver image and the size of the face region. For example, the posture detection unit 31 multiplies the reciprocal of the ratio of the size of the face region detected from the driver image to the reference size of the face region by the reference distance to obtain the distance in the optical axis direction of the driver monitor camera 2, and sets it as the position of the driver's face in the optical axis direction. Hereinafter, the optical axis direction of the driver monitor camera 2 is referred to as the depth direction. The reference size of the face region can be the standard size of the face region when the driver's head is located at a predetermined reference distance from the driver monitor camera 2. The reference size and the size of the face region here can be the horizontal width of the face region on the driver image. In addition, the posture detection unit 31 finds the intersection position where the plane orthogonal to the depth direction at a position separated by the depth direction distance from the driver monitor camera 2 to the driver's face intersects with the azimuth from the driver monitor camera 2 corresponding to the position of the center of gravity of the face region. Then, the posture detection unit 31 sets the horizontal component of the intersection position as the position of the driver's face in the horizontal direction, and sets the vertical component of the intersection position as the position of the driver's face in the height direction.

[0029] The posture detection unit 31 notifies the abnormal posture determination unit 33 of the detected position and orientation of the driver's face. Furthermore, the posture detection unit 31 notifies the heart rate detection unit 32 of the position and range of the face region detected from the driver image.

[0030] The heart rate detection unit 32 detects the driver's heart rate based on a plurality of time-series driver images obtained in the most recent predetermined period. For example, the heart rate detection unit 32 detects the driver's heart rate based on a method for detecting the heart rate based on an image pulse wave. In this case, the heart rate detection unit 32 calculates the average luminance value for each face region of the plurality of driver images. When the processing by the heart rate detection unit 32 and the processing by the posture detection unit 31 are executed in parallel, or when the processing by the heart rate detection unit 32 is performed prior to the processing by the posture detection unit 31, the heart rate detection unit 32 may detect the face region from the driver image by executing the same processing as the detection of the face region by the posture detection unit 31. Further, the heart rate detection unit 32 calculates the power spectrum of the image pulse wave by performing a fast Fourier transform on the waveform of the image pulse wave represented by the temporal change of the average luminance value. Then, the heart rate detection unit 32 specifies the frequency that becomes the maximum value in the calculated power spectrum as the frequency corresponding to the heart rate, and detects the number of cycles per minute corresponding to the specified frequency as the heart rate.

[0031] Furthermore, the heart rate detection unit 32 calculates the average value of the driver's heart rate detected at predetermined intervals after the ignition switch of the vehicle 10 is turned on. At this time, the heart rate detection unit 32 may calculate the average value of the heart rate by using a moving average or an averaging process using a forgetting factor.

[0032] Also, the heart rate detection unit 32 may calculate the variance of the driver's heart rate detected at predetermined intervals after the ignition switch of the vehicle 10 is turned on.

[0033] The heart rate detection unit 32 notifies the detected driver's heart rate and the average value of the heart rate to the heart rate abnormality determination unit 34. Further, when the variance of the driver's heart rate is calculated, the heart rate detection unit 32 also notifies the variance to the heart rate abnormality determination unit 34.

[0034] The abnormal posture determination unit 33 determines whether the detected posture of the driver is a predetermined abnormal posture based on the detection results of the position and orientation of the driver's face notified from the posture detection unit 31.

[0035] The predetermined abnormal postures are, for example, a slumped posture, a bowed posture, a supine posture, a shrimp-like posture, a posture where only the neck is tilted horizontally, a posture where the whole body is tilted horizontally, or a posture of leaning sideways. For each type of such abnormal posture, abnormal determination conditions represented by a combination of respective ranges of the face position and the face orientation, or a range of either the face position or the face orientation are set and stored in advance in the memory 22.

[0036] For example, the abnormal determination conditions for the slumped posture are set as follows. · The face position is 200 mm or more closer to the driver monitor camera 2 (the front side) than the reference position in the depth direction, and 180 mm or more lower than the reference position in the height direction, and · The orientation of the face in the pitch direction is inclined downward by 30° or more from the reference direction (for example, the front direction of the vehicle 10). Also, the abnormal determination conditions for the bowed posture are set as follows. · The orientation of the face in the pitch direction is inclined downward by 20° or more from the reference direction. Furthermore, the abnormal determination conditions for the supine posture are set as follows. · The face position is 100 mm or more farther from the driver monitor camera 2 (the rear side) than the reference position in the depth direction, and · The orientation of the face in the pitch direction is inclined upward by 20° or more from the reference direction. Furthermore, the abnormal determination conditions for the posture where only the neck is tilted horizontally are set as follows. · The orientation of the face in the roll direction is inclined by 30° or more to either the left or the right from the vertical direction. Furthermore, the abnormal determination conditions for the posture where the whole body is tilted horizontally are set as follows. · The position of the face is shifted horizontally by 200 mm or more to either the left or right from the reference position, and · The orientation of the face in the roll direction is inclined by 15° or more from the vertical direction with respect to the direction in which the position of the face is shifted to either the left or right.

[0037] These abnormal determination conditions are just examples, and other abnormal determination conditions may be set for each abnormal posture.

[0038] Among these abnormal postures, the prone posture and the posture where only the neck is tilted are postures that are difficult to assume when the driver is in a normal state, and there is a very high possibility that some abnormality has occurred to the driver. Conversely, among these abnormal postures, postures other than the prone posture and the posture where only the neck is tilted are postures that can be assumed even when the driver is relaxed. Therefore, hereinafter, for the sake of convenience of explanation, among the above abnormal postures, postures other than the prone posture and the posture where only the neck is tilted, that is, postures that the driver can assume without any abnormality, are referred to as the first abnormal postures. On the other hand, the prone posture and the posture where only the neck is tilted, that is, postures that are presumed to have some abnormality in the driver, are referred to as the second abnormal postures.

[0039] Whenever the abnormal posture determination unit 33 is notified of the position and orientation of the driver's face from the posture detection unit 31, it determines whether the position and orientation of the face are included in the abnormal determination conditions for any abnormal posture. And when the state where the position or orientation of the driver's face is included in the abnormal determination conditions for any abnormal posture continues for a time threshold (for example, 1 second to 3 seconds) or more, the abnormal posture determination unit 33 determines that the driver's posture is the abnormal posture corresponding to the abnormal determination conditions.

[0040] When the abnormal posture determination unit 33 determines that the driver's posture is any abnormal posture, it notifies the heartbeat abnormal determination unit 34, the notification processing unit 35, and the vehicle control unit 36 of the determination result and the type of the abnormal posture determined to be assumed by the driver.

[0041] When the detected posture of the driver is the first abnormal posture (for example, a bowed posture or a lying-on-side posture, etc.), the heartbeat abnormality determination unit 34 determines the presence or absence of an abnormality in the driver's heartbeat based on the heartbeat rate of the driver detected by the heartbeat detection unit 32.

[0042] To determine the presence or absence of an abnormality in the driver's heartbeat, when the detected posture of the driver is the first abnormal posture, the heartbeat abnormality determination unit 34 calculates a deviation index value representing the degree of deviation of the latest detected heartbeat rate from the average value of the heartbeat rates. The heartbeat abnormality determination unit 34 calculates, for example, the absolute value of the difference between the latest heartbeat rate and the average value of the heartbeat rates or the square of the difference as the deviation index value. Further, when the variance of the driver's heartbeat rate has been calculated, the heartbeat abnormality determination unit 34 may calculate the Mahalanobis distance between the latest heartbeat rate and the average value of the heartbeat rates as the deviation index value. Then, the heartbeat abnormality determination unit 34 compares the deviation index value with a predetermined abnormality determination threshold value. When the deviation index value is greater than the abnormality determination threshold value, the heartbeat abnormality determination unit 34 determines that there is an abnormality in the driver's heartbeat.

[0043] Alternatively, when the latest detected heartbeat rate detected by the heartbeat abnormality determination unit 34 is outside the preset normal range of the heartbeat rate, it may be determined that there is an abnormality in the driver's heartbeat. For example, when the latest detected heartbeat rate is lower than the lower limit value (for example, 40) of the normal range, the heartbeat abnormality determination unit 34 determines that there is an abnormality in the driver's heartbeat.

[0044] On the other hand, when the deviation index value is less than or equal to the abnormality determination threshold value and the latest detected heartbeat rate is included in the normal range of the heartbeat rate, the heartbeat abnormality determination unit 34 determines that there is no abnormality in the driver's heartbeat.

[0045] Alternatively, when the duration during which the deviation index value becomes greater than the abnormality determination threshold value or the duration during which the detected heartbeat rate is outside the normal range of the heartbeat rate is equal to or longer than a predetermined time length (for example, several seconds), the heartbeat abnormality determination unit 34 may determine that there is an abnormality in the driver's heartbeat.

[0046] The heart rate abnormality determination unit 34 notifies the notification processing unit 35 and the vehicle control unit 36 of the determination result regarding the presence or absence of an abnormality in the driver's heart rate.

[0047] When the notification processing unit 35 receives from the heart rate abnormality determination unit 34 a determination result indicating that there is an abnormality in the driver's heart rate, or receives from the abnormal posture determination unit 33 a determination result indicating that the driver's posture is the second abnormal posture, the notification processing unit 35 performs a predetermined warning process. For example, the notification processing unit 35 causes the speaker included in the notification device 3 to emit an audio signal or a warning sound requesting the driver to assume a driving posture. Alternatively, the notification processing unit 35 causes the display included in the notification device 3 to display a warning message requesting the driver to assume a driving posture. Or, the notification processing unit 35 causes the vibrator included in the notification device 3 to vibrate.

[0048] After the notification processing unit 35 performs a warning process of requesting the driver to assume a driving posture via the notification device 3, if the notification processing unit 35 receives from the abnormal posture determination unit 33 a determination result indicating that the driver's posture does not correspond to any abnormal posture, the notification processing unit 35 stops the execution of the warning process.

[0049] FIG. 4 is a diagram for explaining the outline of the driver monitoring process according to the present embodiment. In this example, the driver's face 400 is facing downward by 20° or more from the reference direction indicated by the arrow 401, and the driver's posture is detected as a bowed posture. At this time, if it is not determined that there is an abnormality in the heart rate, it is highly likely that the driver is simply taking a non-driving action, so no warning is notified. On the other hand, if it is determined that there is an abnormality in the heart rate, there may be some abnormality that prevents the driver from properly driving the vehicle 10. Therefore, a warning is notified to the driver.

[0050] The vehicle control unit 36 controls the vehicle 10 according to the driving control level applied to the vehicle 10 until it receives a determination result from the heartbeat abnormality determination unit 34 that there is an abnormality in the driver's heartbeat, or a determination result from the abnormal posture determination unit 33 that the driver's posture is the second abnormal posture. When the driving control level applied to the vehicle 10 is a driving control level in which the driver is not involved in the driving of the vehicle 10, the vehicle control unit 36 controls the vehicle 10 to travel along the own lane in which the vehicle 10 is traveling. To that end, the vehicle control unit 36 detects a lane dividing line that divides the own lane and the adjacent lane from an image generated by an external camera, and moving objects such as other vehicles traveling around the vehicle 10. Then, the vehicle control unit 36 estimates the position and posture of the vehicle 10 by comparing the detected lane dividing line with the map information. Then, based on the estimation result of the position and posture of the vehicle 10 and the detection result of each moving object around the vehicle 10, the vehicle control unit 36 controls the vehicle 10 so that the vehicle 10 does not collide with each moving object and travels along the own lane.

[0051] Also, when the vehicle control unit 36 continuously receives a determination result from the heartbeat abnormality determination unit 34 that there is an abnormality in the driver's heartbeat for a certain period, the vehicle control unit 36 controls the vehicle 10 to make an emergency stop. Similarly, when the vehicle control unit 36 continuously receives a determination result from the abnormal posture determination unit 33 that the driver's posture is the second abnormal posture for a certain period, the vehicle control unit 36 controls the vehicle 10 to make an emergency stop. Note that the vehicle control unit 36 may control the vehicle 10 to make an emergency stop immediately when it receives a determination result from the heartbeat abnormality determination unit 34 that there is an abnormality in the driver's heartbeat, or a determination result that the driver's posture is the second abnormal posture. At that time, the vehicle control unit 36 may move the vehicle 10 to the road shoulder and then stop it based on the estimation result of the position and posture of the vehicle 10, the detection result of each moving object around the vehicle 10, and the map information.

[0052] FIG. 5 is an operation flowchart of vehicle control processing including driver monitoring processing executed by the processor 23. The processor 23 may execute vehicle control processing according to the following operation flowchart at predetermined intervals.

[0053] The posture detection unit 31 of the processor 23 detects the position and orientation of the driver's face as the driver's posture based on the driver image (step S101). Also, the heartbeat detection unit 32 of the processor 23 detects the driver's heartbeat based on a plurality of time-series driver images obtained in a recent predetermined period (step S102).

[0054] The abnormal posture determination unit 33 of the processor 23 determines whether the driver's posture is a first abnormal posture (step S103). If the driver's posture is a first abnormal posture (step S103 - Yes), the heartbeat abnormality determination unit 34 of the processor 23 determines whether there is an abnormality in the detected driver's heartbeat based on the detected driver's heartbeat (step S104). If there is an abnormality in the driver's heartbeat (step S104 - Yes), the notification processing unit 35 of the processor 23 notifies the driver of a warning requesting to take a driving posture via the notification device 3 (step S105). Further, the vehicle control unit 36 of the processor 23 controls the vehicle 10 to cause the vehicle 10 to make an emergency stop (step S106).

[0055] On the other hand, in step S104, if no abnormality in the driver's heartbeat is detected (step S104 - No), the vehicle control unit 36 controls the vehicle 10 according to the driving control level applied to the vehicle 10 (step S107).

[0056] Also, in step S103, when the driver's posture is different from the first abnormal posture (step S103 - No), the abnormal posture determination unit 33 determines whether the driver's posture is the second abnormal posture (step S108). When the driver's posture is the second abnormal posture (step S108 - Yes), the processor 23 executes the processes after step S105. On the other hand, when the driver's posture is different from the second abnormal posture (step S108 - No), the processor 23 executes the process of step S107.

[0057] After step S106 or step S107, the processor 23 ends the vehicle control process. Note that the processor 23 may change the execution order of the process of step S101 and the process of step S102, or may execute the process of step S101 and the process of step S102 in parallel. Further, the processor 23 may execute the process of step S102, that is, the process of detecting the heart rate, only when it is determined that the driver's posture is the first abnormal posture. Thereby, the calculation load of the processor 23 is reduced, and the hardware resources used for the vehicle control process can be suppressed.

[0058] As described above, when the posture of the driver detected from the driver image is a predetermined abnormal posture, this driver monitoring device determines whether there is an abnormality in the driver's heartbeat. And when it is determined that there is an abnormality in the driver's heartbeat, this driver monitoring device notifies the driver of a warning via a notification device provided in the vehicle interior. In this way, this driver monitoring device not only detects an abnormal posture of the driver from the driver image, but also detects an abnormality in the driver's heartbeat. Therefore, this driver monitoring device can suppress false detection that an abnormality has occurred in the driver due to the driver performing non-driving actions even though no abnormality has occurred in the driver. Thus, this driver monitoring device can appropriately detect an abnormality that has occurred in the driver. As a result, this driver monitoring device can suppress giving unnecessary warnings to the driver, so that the convenience of the driver can be improved. Furthermore, since this driver monitoring device performs both detection of the driver's posture from the driver image and determination of whether there is an abnormality in the driver's heartbeat, it does not require a sensor for obtaining driver information other than a camera for photographing the driver. Therefore, this driver monitoring device can simplify the system configuration.

[0059] According to a modification, even when it is determined that the posture of the driver is a second abnormal posture, the heartbeat abnormality determination unit 34 may determine whether there is an abnormality in the driver's heartbeat. And the notification processing unit 35 may notify the driver of a warning via the notification device 3 only when it is determined that there is an abnormality in the driver's heartbeat even when it is determined that the posture of the driver is a second abnormal posture.

[0060] Also, a computer program that realizes the functions of the processor 23 of the ECU 4 according to the above-described embodiment or modification may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0061] As described above, those skilled in the art can make various modifications according to the implemented forms within the scope of the present invention.

Explanation of Reference Numerals

[0062] 1 Vehicle control system 2 Driver monitoring camera 3 Notification device 4 Electronic control unit (driver monitoring device) 10 Vehicle 21 Communication interface 22 Memory 23 Processor 31 Posture detection unit 32 Heartbeat detection unit 33 Abnormal posture determination unit 34 Heartbeat abnormality determination unit 35 Notification processing unit 36 Vehicle control unit

Claims

1. A posture detection unit that detects the posture of the driver of the vehicle based on at least any one of a plurality of images in the vehicle interior of the vehicle generated in time series by an imaging unit provided in the vehicle; A heart rate detection unit that detects the heart rate of the driver based on the plurality of images; An abnormal posture determination unit that determines whether the detected posture of the driver corresponds to either a first abnormal posture or a second abnormal posture; A heart rate abnormality determination unit that determines the presence or absence of an abnormality in the driver's heart rate based on the detected heart rate when the detected posture of the driver is the first abnormal posture; A notification processing unit that notifies the driver of a warning via a notification device provided in the vehicle interior when it is determined that there is an abnormality in the heart rate, or when the detected posture of the driver is the second abnormal posture; A driver monitoring device having the above.

2. The driver monitoring device according to claim 1, wherein the heart rate detection unit detects the heart rate of the driver only when it is determined that the detected posture of the driver is the first abnormal posture.

3. Detect the posture of the driver of the vehicle based on at least any one of a plurality of images in the vehicle interior of the vehicle generated in time series by an imaging unit provided in the vehicle, Detect the heart rate of the driver based on the plurality of images, Determine whether the detected posture of the driver corresponds to either a first abnormal posture or a second abnormal posture, When the detected posture of the driver is the first abnormal posture, determine the presence or absence of an abnormality in the driver's heart rate based on the detected heart rate, When it is determined that there is an abnormality in the heart rate, or when the detected posture of the driver is the second abnormal posture, notify the driver of a warning via a notification device provided in the vehicle interior. A driver monitoring method including the above.

4. Detect the posture of the driver of the vehicle based on at least any one of a plurality of images in the vehicle interior of the vehicle generated in time series by an imaging unit provided in the vehicle, Detect the heart rate of the driver based on the plurality of images, Determine whether the detected posture of the driver corresponds to either a first abnormal posture or a second abnormal posture, When the detected posture of the driver is the first abnormal posture, determine the presence or absence of an abnormality in the driver's heart rate based on the detected heart rate, When it is determined that there is an abnormality in the heartbeat, or when the detected posture of the driver is the second abnormal posture, a warning is notified to the driver via a notification device provided in the vehicle interior. This is a computer program for driver monitoring to be executed by a processor mounted on the vehicle.

Images