Monitoring devices

Abstract

This device eliminates the possibility of mistakenly monitoring people other than the driver and provides a monitoring system that warns the driver if the driver's face cannot be detected. [Solution] The monitoring device acquires image data of the interior of the vehicle using a camera installed inside the vehicle. The face detection unit detects the faces of people present inside the vehicle from the image data. The monitoring device determines whether the detected face is a face viewed from a predetermined side, either the right face viewed from the front right or the left face viewed from the front left. If the monitoring device determines that the detected face is a face viewed from the predetermined side, it assumes that the detected face is the driver's face and monitors the driver's status. If the monitoring device determines that the detected face is not a face viewed from the predetermined side, it outputs a warning.

Description

Technical Field

[0001] The present invention relates to a monitoring device.

Background Art

[0002] Driver monitoring systems have been conventionally known. A driver monitoring system captures an image of a driver with a camera, monitors, in real time, for example, the driver's line-of-sight direction and the driver's face orientation from the captured driver image, and detects driver inattention, fatigue, drowsiness or the like based on the line-of-sight direction, face orientation and the like, and outputs a warning at an early stage.

[0003] Further, a mirror-integrated driver monitoring system in which a camera is provided in a rearview mirror is also known. Since the mirror-integrated driver monitoring system is provided separately from the vehicle system, the processing load of the vehicle system can be reduced. Furthermore, the mirror-integrated driver monitoring system can be externally attached to a vehicle that is not equipped with a driver monitoring system.

[0004] Patent Document 1 describes a technique for distinguishing between a driver and an occupant in an image captured inside a vehicle based on coordinates on the image. Patent Document 2 describes a technique for preventing an occupant other than the driver from being erroneously recognized as the driver's face by changing the size or position of a face recognition region based on the extension length of a seatbelt of a seat other than the driver's seat. Patent Document 3 describes a technique for identifying an occupant who operates an operation device inside the vehicle by detecting the line of sight, face orientation, and hands of the vehicle occupant.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] Incidentally, the rearview mirror's position is adjusted by the driver as needed so that the image of the area behind the vehicle is reflected in the mirror from the driver's perspective. However, the rearview mirror's position may also change unintentionally due to, for example, contact with a person. Therefore, in mirror-integrated driver monitoring systems, the image captured by the camera installed in the rearview mirror may include the faces of people other than the driver, or even multiple people's faces. In such cases, mirror-integrated driver monitoring systems could mistakenly monitor people other than the driver.

[0007] The present invention has been made in view of the above, and provides a monitoring device that eliminates the possibility of mistakenly monitoring a person other than the driver and can warn the driver if the driver's face cannot be detected. [Means for solving the problem]

[0008] The monitoring device according to the present invention comprises an acquisition unit, a face detection unit, a determination unit, a monitoring unit, and a warning unit. The acquisition unit acquires image data of the interior of a vehicle captured by a camera installed inside the vehicle. The face detection unit detects a person's face present inside the vehicle from the image data. The determination unit determines whether the detected face is a face viewed from a predetermined side, either the right face viewed from the front right or the left face viewed from the front left. If the monitoring unit determines that the detected face is a face viewed from the predetermined side, it assumes the detected face is the driver's face and monitors the driver's status. If the warning unit determines that the detected face is not a face viewed from the predetermined side, it outputs a warning. [Effects of the Invention]

[0009] The monitoring device according to the present invention eliminates the possibility of mistakenly monitoring people other than the driver, and can warn the driver if the driver's face cannot be detected. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows the exterior of a vehicle according to this embodiment. [Figure 2] Figure 2 shows the configuration of the driver monitoring system. [Figure 3] Figure 3 shows the functional configuration of the ECU according to the embodiment. [Figure 4] Figure 4 is a flowchart showing the processing flow of the ECU according to the embodiment. [Figure 5] Figure 5 shows an example of the left side of the face. [Figure 6] Figure 6 shows an example of the right side of the face. [Modes for carrying out the invention]

[0011] The vehicle 10 according to this embodiment will be described below with reference to the drawings.

[0012] Figure 1 shows the external appearance of vehicle 10 according to this embodiment.

[0013] Vehicle 10 is equipped with a mirror-integrated driver monitoring system 20. The mirror-integrated driver monitoring system 20 is a monitoring device that monitors the driver's condition. The mirror-integrated driver monitoring system 20 uses a camera 24 installed in the rearview mirror 22 to capture images of the driver seated in the driver's seat. The mirror-integrated driver monitoring system 20 monitors the driver's gaze direction and the direction of the driver's face in real time from the captured image data, and detects, for example, driver inattention, fatigue, or drowsiness based on the gaze direction and face direction. When the mirror-integrated driver monitoring system 20 detects driver inattention, fatigue, or drowsiness, it notifies the driver of the detection information. Such a mirror-integrated driver monitoring system 20 can notify the driver of driver inattention, fatigue, or drowsiness at an early stage.

[0014] The rearview mirror 22 has a mirror surface that reflects an image of the area behind the vehicle 10. The rearview mirror 22 is positioned at the front of the interior of the vehicle 10, in a location where a person seated in the driver's seat of the vehicle 10 can see the mirror surface. The rearview mirror 22 is adjustable in position. This allows the rearview mirror 22 to allow the driver to see objects behind the vehicle 10 through the mirror surface, regardless of the driver's height or posture.

[0015] Figure 2 shows the configuration of the driver monitoring system 20.

[0016] The driver monitoring system 20 includes a camera 24, an ECU (Electronic Control Unit) 26, a display device 28, and a speaker 30.

[0017] Camera 24 is mounted on the rearview mirror 22. Camera 24 is fixed to the rearview mirror 22 and captures images of the driver seated in the driver's seat from the rearview mirror 22. Camera 24 provides the image data of the driver to the ECU 26. Camera 24 may be mounted in a location other than the rearview mirror 22, as long as it is capable of capturing images of the driver's face.

[0018] The ECU 26 is incorporated in the rearview mirror 22. The ECU 26 is an information processing apparatus that executes a pre-stored program. The ECU 26 acquires image data from the camera 24. The ECU 26 detects the driver's line-of-sight direction, the driver's face orientation, and the like from the acquired image data. The ECU 26 detects the driver's state, for example, driver inattention, fatigue, drowsiness, and the like, based on the corrected line-of-sight direction, face orientation, and the like. Then, when detecting driver inattention, fatigue, drowsiness, or the like, the ECU 26 outputs detection information from the display device 28 or the speaker 30.

[0019] The display device 28 is provided outside the rearview mirror 22. The display device 28 may be, for example, a monitor device provided in a navigation system, or a monitor device provided on a control panel of the vehicle 10. The display device 28 outputs the detection information output from the ECU 26.

[0020] The speaker 30 may be provided outside the rearview mirror 22, or may be provided on the rearview mirror 22. When the speaker 30 is provided outside the rearview mirror 22, it may be, for example, a speaker device provided in a navigation system, or a speaker device provided on a control panel of the vehicle 10. The speaker 30 outputs the detection information output from the ECU 26.

[0021] Figure 3 is a diagram showing a functional configuration of the ECU 26 according to the embodiment.

[0022] By executing the program, the ECU 26 functions as an acquisition unit 42, a face detection unit 44, a determination unit 46, a monitoring unit 48, an output unit 50, and a warning unit 52.

[0023] The acquisition unit 42 acquires image data obtained by imaging the interior of the vehicle 10 with the camera 24.

[0024] The face detection unit 44 detects human faces present inside the vehicle 10 from the image data acquired by the acquisition unit 42. If the image data contains multiple human faces, the face detection unit 44 detects one face from among the multiple faces located around the driver's seat of the vehicle 10.

[0025] The determination unit 46 determines whether the face detected by the face detection unit 44 is a face viewed from a predetermined side, either the right face viewed from the front right or the left face viewed from the front left. For example, if the camera 24 is mounted on the rearview mirror 22 and the driver's seat where the driver sits is located on the right side of the front row of the vehicle 10, the determination unit 46 determines whether the detected face is a left face. Also, for example, if the camera 24 is mounted on the rearview mirror 22 and the driver's seat is located on the left side of the front row of the vehicle 10, the determination unit 46 determines whether the detected face is a right face. In this way, the determination unit 46 can determine whether the face it has detected is the driver's face.

[0026] Furthermore, the determination unit 46 may further determine whether the detected face is the face of a person sitting in the driver's seat of the vehicle 10. For example, the determination unit 46 performs image analysis on the image data to detect feature points representing the interior of the vehicle 10, such as seats and headrests, and determines that the detected face is the face of a person sitting in the driver's seat of the vehicle 10 based on the relationship between the position of the feature points and the position of the detected face.

[0027] If the monitoring unit 48 determines that the face detected by the judgment unit 46 is a face viewed from a predetermined side, it assumes the detected face is the driver's face and monitors the driver's condition. Alternatively, if the judgment unit 46 determines that the detected face is a face viewed from a predetermined side and is the face of a person seated in the driver's seat of the vehicle 10, the monitoring unit 48 may assume the detected face is the driver's face and monitor the driver's condition. For example, the monitoring unit 48 monitors the driver's gaze direction and the orientation of the driver's face in real time from the captured image data and detects driver inattention, fatigue, or drowsiness based on the gaze direction and face orientation.

[0028] The output unit 50 outputs detection information from the display device 28 or speaker 30 when the monitoring unit 48 detects driver inattention, fatigue, drowsiness, etc. This allows the output unit 50 to inform the driver that inattention, fatigue, drowsiness, etc., has occurred.

[0029] The warning unit 52 outputs a warning from the display device 28 or speaker 30 if the determination unit 46 determines that the detected face is not a face viewed from a predetermined side. Alternatively, the warning unit 52 may output a warning from the display device 28 or speaker 30 if the determination unit 46 determines that the detected face is not a face viewed from a predetermined side, or that the detected face is not the face of a person seated in the driver's seat of the vehicle 10. This allows the warning unit 52 to notify the driver of a warning, for example, if the rearview mirror 22 is not in the proper position and the camera 24 fixed to the rearview mirror 22 is not capturing the driver's face. This allows the driver to adjust the position of the rearview mirror 22 so that the camera 24 can properly capture the driver's face.

[0030] Figure 4 is a flowchart showing the processing flow of the ECU26 according to the embodiment. Figure 5 is a diagram showing an example of the left side of the face. Figure 6 is a diagram showing an example of the right side of the face.

[0031] The driver monitoring system 20 executes the processes from S21 to S26, for example, at predetermined intervals or whenever a predetermined event occurs, such as a change in the position of the rearview mirror 22.

[0032] First, in S21, the ECU 26 acquires image data of the interior of the vehicle 10 captured by the camera 24. Then, the ECU 26 detects the faces of people present in the interior of the vehicle 10 from the acquired image data. If the image data contains the faces of multiple people, the ECU 26 detects one face from among the multiple faces, specifically one face located around the driver's seat of the vehicle 10. For example, the ECU 26 performs image analysis on the image data to detect feature points representing the interior of the vehicle 10, such as seats and headrests, and then detects one face around the driver's seat based on its positional relationship with the location of the feature points.

[0033] Next, in S22, the ECU26 determines whether or not a face was detected from the image data. If the ECU26 cannot detect a face from the image data (No in S22), it determines that it is unknown whether or not there is a person inside the vehicle 10 and proceeds to S26. If the ECU26 can detect a face from the image data (Yes in S22), it proceeds to S23.

[0034] In S23, the ECU 26 determines whether the detected face is a face viewed from a predetermined side, either the right face as seen from the front right or the left face as seen from the front left. For example, if the camera 24 is mounted on the rearview mirror 22 and the driver's seat where the driver sits is located on the right side of the front row of the vehicle 10, the face viewed from the predetermined side is the left face. Therefore, in this case, the ECU 26 determines whether the detected face is the left face. Also, for example, if the camera 24 is mounted on the rearview mirror 22 and the driver's seat is located on the left side of the front row of the vehicle 10, the face viewed from the predetermined side is the right face. Therefore, in this case, the ECU 26 determines whether the detected face is the right face.

[0035] For example, the ECU26 may determine whether the detected face is a right-handed or left-handed face based on the positional relationship of feature points such as the eyes and nose on the detected face. For example, as shown in Figure 5, the ECU26 determines that the detected face is a left-handed face if the eyes and nose are located to the left of the center of the face. Also, for example, as shown in Figure 6, the ECU26 determines that the detected face is a right-handed face if the eyes and nose are located to the right of the center of the face.

[0036] If ECU26 determines that the detected face is not a face viewed from a predetermined side, either the right or left side (No in S23), it assumes the detected face is the face of someone other than the driver and proceeds to S26. If ECU26 determines that the detected face is a face viewed from a predetermined side, either the right or left side (Yes in S23), it proceeds to S24.

[0037] In S24, the ECU26 determines whether the detected face is the face of a person sitting in the driver's seat of vehicle 10. For example, the ECU26 analyzes the image data to detect feature points representing the interior of the vehicle 10, such as seats and headrests, and determines whether the detected face is the face of a person sitting in the driver's seat of vehicle 10 based on the relationship between the position of the feature points and the position of the detected face.

[0038] If ECU26 determines that the detected face is not the face of the person sitting in the driver's seat of vehicle 10 (No in S24), it proceeds to S26, assuming that the detected face is the face of someone other than the driver. If ECU26 determines that the detected face is the face of the person sitting in the driver's seat of vehicle 10 (Yes in S24), it proceeds to S25.

[0039] In S25, the ECU 26 assumes that the detected face is the driver's face and continuously performs monitoring processing to monitor the driver's state. For example, the ECU 26 monitors the driver's gaze direction and the orientation of the driver's face in real time from the captured image data and detects driver inattention, fatigue, or drowsiness based on the gaze direction and face orientation. If the ECU 26 detects driver inattention, fatigue, or drowsiness, it outputs detection information from the display device 28 or speaker 30. The ECU 26 then returns to processing in S21 while continuing the monitoring processing, and repeats the process from S21 after a predetermined time or after a predetermined event occurs.

[0040] In S26, the ECU26 outputs a warning from the display device 28 or speaker 30 indicating that the driver status monitoring process is not being performed properly. After outputting the warning, the ECU26 returns to S21 and repeats the process from S21 after a predetermined time or after a predetermined event occurs. In this case, the ECU26 may continue to output the warning until the monitoring process in S25 is properly started.

[0041] As described above, the driver monitoring system 20 according to this embodiment monitors the driver's status if it determines that the face detected from the image data captured by the camera 24 is a face viewed from a predetermined side, and outputs a warning if it determines that the detected face is not a face viewed from the predetermined side. In this way, the driver monitoring system 20 according to this embodiment eliminates the possibility of mistakenly monitoring a person other than the driver and can warn the driver if the driver's face cannot be detected.

[0042] Furthermore, in this embodiment, the camera 24 is fixedly mounted on the rearview mirror 22. Therefore, the position of the rearview mirror 22 may be adjusted, or a person or other object may come into contact with the rearview mirror 22, changing its position. Even in such cases, the driver monitoring system 20 according to this embodiment will warn the driver if it cannot detect the driver's face, allowing the driver to readjust the position of the rearview mirror 22.

[0043] Furthermore, the driver monitoring system 20 according to this embodiment monitors the driver's status if it determines that the detected face is a face viewed from a predetermined side and is the face of a person seated in the driver's seat of the vehicle 10. If the detected face is not a face viewed from a predetermined side, or if the detected face is not the face of a person seated in the driver's seat of the vehicle 10, a warning is output. As a result, the driver monitoring system 20 according to this embodiment eliminates the possibility of mistakenly monitoring the faces of other people in the vehicle 10's interior who are looking in the same direction as the driver, and ensures reliable monitoring of the driver's status.

[0044] Furthermore, the driver monitoring system 20 according to this embodiment detects one face in the vicinity of the driver's seat of the vehicle 10 when the image data includes the faces of multiple people. As a result, the driver monitoring system 20 according to this embodiment can reliably identify the driver from among multiple people and reliably monitor the driver's condition, even when multiple people are present inside the vehicle 10.

[0045] Although embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. Various modifications can be made to the embodiments. [Explanation of Symbols]

[0046] 10...Vehicle, 20...Driver monitoring system, 22...Rearview mirror, 24...Camera, 26...ECU, 28...Display device, 30...Speaker, 42...Acquisition unit, 44...Face detection unit, 46...Decision unit, 48...Monitoring unit, 50...Output unit, 52...Warning unit

Claims

1. An acquisition unit that acquires image data of the interior of a vehicle by a camera installed inside the vehicle, A face detection unit that detects a person's face inside the vehicle from the aforementioned image data, A determination unit that determines whether the detected face is a face viewed from a predetermined side, either the right face viewed from the front right or the left face viewed from the front left. If it is determined that the detected face is a face viewed from the predetermined side, the monitoring unit monitors the driver's status, assuming that the detected face is the driver's face. A warning unit outputs a warning if it is determined that the detected face is not a face viewed from the predetermined side. A monitoring device equipped with the following features.

2. The determination unit determines whether the detected face is the face of a person sitting in the driver's seat of the vehicle. If the monitoring unit determines that the detected face is a face viewed from the predetermined side and is the face of a person seated in the driver's seat of the vehicle, it determines that the detected face is the face of the driver and monitors the driver's status. The warning unit outputs the warning if the detected face is not a face viewed from the predetermined side, or if the detected face is not the face of a person sitting in the driver's seat of the vehicle. The monitoring device according to claim 1.

3. The face detection unit detects one face in the vicinity of the driver's seat of the vehicle if the image data contains the faces of multiple people. The monitoring device according to claim 1.

4. The aforementioned camera is fixed to the rearview mirror, The rearview mirror is positioned at the front of the vehicle's interior, in a location where a person seated in the driver's seat of the vehicle can see the mirror surface, and its position can be adjusted. The monitoring device according to claim 1.

Images