Driver monitoring device and program, and medium

Abstract

Provided is a driver monitoring device that is capable of being retrofitted to a vehicle. The driver monitoring device has an image capturing unit for capturing an image including the driver of the vehicle, a control unit, and an output unit. The control unit identifies the line-of-sight direction of the driver from the image captured by the image capturing unit, and determines whether the identified line-of-sight direction has been oriented below a preset imaginary line continuously for longer than a prescribed time. If it has been determined that the line-of-sight direction has been oriented below the imaginary line continuously for longer than the prescribed time, the output unit outputs an alert to the driver.

Description

Driver monitoring device, program and medium

[0001] The present invention relates to a driver monitoring device, a program, and a medium for monitoring a driver who drives a vehicle, for example.

[0002] There is a device known as a drive recorder that is installed in a vehicle and has a function of capturing images of the outside of the vehicle and recording the latest images for a predetermined period of time. Some drive recorders are also equipped with a camera for capturing images of the inside of the vehicle. Patent Document 1 describes a driving situation monitoring device that receives driving situation data including an image of a driver for authentication captured by the drive recorder, vehicle information, position information, and acceleration information, and records the driver's identification information and the driving situation data in association with each other.

[0003] Patent Document 1 further describes that the driving situation monitoring device generates dangerous driving data in response to the occurrence of an event and transmits warning information to the drive recorder.

[0004] Japanese Patent Application Laid-Open No. 2022-111162

[0005] The driving condition monitoring device described in Patent Document 1 monitors the vehicle status such as the vehicle speed, acceleration, and position. Although it photographs the driver, the image is only used to identify the driver and is not used to monitor the driver's condition.

[0006] The present invention has been made in view of the above-mentioned conventional examples, and has an object to contribute to safer vehicle operation by monitoring the driver's behavior.

[0007] In order to achieve the above object, the present invention has the following configuration: According to one aspect of the present invention, there is provided a driver monitoring device that can be retrofitted to a vehicle, comprising: an imaging means for capturing an image including the driver of the vehicle; a control means; and an output means, wherein the control means identifies the driver's line of sight from the image captured by the imaging means; determines whether the identified line of sight is directed below a predetermined virtual line continuously for more than a predetermined time; and when it is determined that the line of sight is directed below the virtual line continuously for more than the predetermined time, outputs a warning to the driver by the output means.

[0008] According to the present invention, by monitoring the driver's condition, it is possible to contribute to even safer vehicle operation.

[0009] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0010] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and together with the description are used to explain the principles of the present invention. Fig. 1 is a diagram showing a vehicle according to an embodiment. Fig. 2 is a block diagram of a drive recorder according to an embodiment. Fig. 3 is a diagram explaining a driver's line of sight vector and a virtual line. Fig. 4 is a flowchart of a virtual line setting process according to a first embodiment. Fig. 5 is a flowchart of a driver monitoring process according to a first embodiment. Fig. 6 is a flowchart of a virtual line setting process according to a second embodiment. Fig. 7 is a flowchart (partial) of a driver monitoring process according to a third embodiment. Fig. 8 is a flowchart (partial) of a driver monitoring process according to a fourth embodiment. Fig. 9 is a flowchart (partial) of a driver monitoring process according to a fifth embodiment. Fig. 10 is a flowchart (partial) of a driver monitoring process according to a fifth embodiment.

[0011] [First Embodiment] Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings. Note that the following embodiment does not limit the invention according to the claims, and not all combinations of features described in the embodiment are necessarily essential to the invention. Two or more of the multiple features described in the embodiment may be combined in any manner. Furthermore, the same reference numerals are used for the same or similar configurations, and redundant explanations will be omitted.

[0012] This embodiment relates to an information processing device for detecting the amount of movement of a moving body to which the driver monitoring device is attached. In the following embodiment, a case will be described in which the driver monitoring device is a drive recorder attached to a moving body such as a vehicle. However, the driver monitoring device may be any other device that is attached to a moving body and can capture images of the inside and outside of the moving body. In addition, in the following embodiment, a case will be described in which the moving body is, for example, a vehicle. The vehicle is typically a four-wheeled vehicle, but the invention may be applied to other types of vehicles as long as the vehicle is driven and operated by an occupant.

[0013] An example of the mounting position of a drive recorder will be described with reference to Fig. 1. A drive recorder 101 is mounted on a vehicle 100. The drive recorder 101 is mounted near the rearview mirror on the windshield of the vehicle 100 at a position that does not obstruct the driver's field of view, or on the rearview mirror of the vehicle 100. The mounting position of the drive recorder 101 is not limited to this, and it may be any position that can capture an image of the outside world of the vehicle 100.

[0014] The drive recorder 101 can operate independently, but may also receive power from the vehicle 100. The drive recorder 101 is equipped with an exterior camera 101F that captures the view outside the vehicle, particularly the view ahead of the vehicle, and an interior camera 101R that captures the view inside the vehicle, particularly the driver. The interior camera 101R has a viewing angle large enough to capture an image of the face of the driver seated in the driver's seat 102.

[0015] As will be described later, the drive recorder 101 sets a virtual line 103 indicating the position of the bottom edge of the windshield (shown by a dotted line in the figure). The line where the dashboard 104, located near the front end of the passenger compartment of the vehicle 100, meets the windshield may be curved, but in this embodiment, the virtual line 103 is a straight line to simplify processing and settings. Of course, the virtual line 103 may have a shape that matches the actual bottom edge of the windshield. Since the virtual line 103 is a line that serves as a reference for determining that the driver is looking outside the vehicle (particularly forward) if they are looking above it, if the bottom edge of the windshield is curved as shown in Figure 1, the virtual line 103 may be a straight line extending from the point closest to the front of the vehicle on the curve as a reference in the width direction of the vehicle.

[0016] Example of Drive Radar Configuration Figure 2 shows an example of the control configuration of the drive recorder 101 of this embodiment. This drive recorder is retrofitted to a vehicle and does not require external input signals. However, the power source of the drive recorder 101 depends on the vehicle 100, and the user interface may use a device mounted on the vehicle 100 or another device. Of course, the drive recorder 101 may have an input / output device for providing a user interface. In this embodiment, the drive recorder monitors the driver, and in light of this, the drive recorder 101 is sometimes called a driver monitoring device, or a driving assistance device.

[0017] In FIG. 2 , the drive recorder 101 includes a control unit 200, an in-vehicle camera 101R, and an exterior camera 101F. These cameras are sometimes referred to as a capture unit or image acquisition unit. Camera interfaces (IF) 201 and 202 are signal interfaces with the in-vehicle camera 101R and the exterior camera 101F, respectively. Each camera captures video (moving image) at a predetermined frame rate, for example, 29 fps (frames per second). The captured images are processed by an image processing unit 203, and further necessary processing is performed by a CPU 204, and the resulting video is saved as a video file in an erasable ROM 209. Each video file contains video of a predetermined duration, for example, approximately 30 to 60 seconds. Once the predetermined duration of recording is completed, a new video file is recorded. There are at least two video files. When the erasable ROM 209 runs out of free space, the oldest video file is erased to free up space, and a new video file is recorded there. The captured video can be referenced frame by frame, and in this example, a frame is sometimes referred to as an image. The processing performed by the image processing unit 203 may include, for example, a process of converting a captured image of an object into an image of the object viewed from a different angle, such as projective transformation. The CPU 204 executes a program with a procedure that will be described later, and also has a built-in timer (not shown).

[0018] In order to function as a drive recorder, the drive recorder also includes an acceleration sensor, and when acceleration exceeding a predetermined value is detected, control is performed such as stopping the recording of the video file. However, in this embodiment, the description focuses on the driving assistance function of this drive recorder, and therefore a description of the functions and devices as a drive recorder will be omitted.

[0019] The communication unit 208 provides wired or wireless communication functions. For example, in order to output a warning (described later), the communication unit 208 may be connected to a smartphone or the like and output the warning thereto. Of course, communication for other purposes is also possible.

[0020] The input unit 206 and the output unit 207 may include an input / output unit as a user interface, and the input / output unit may be realized, for example, by a touch panel constituting an operation unit. Alternatively, an interface for connecting to a display audio device provided in the vehicle 100 may be provided. Furthermore, an interface for connecting to a mobile terminal may be provided. The RAM 205 is used as data memory required for the operation of the CPU 204 and image processing unit 203, and as memory for storing programs executed by the CPU 204. The power supply unit 210 is connected to a power source provided by the vehicle 100 and provides power supply power suitable for the drive recorder.

[0021] Overview of Driver Monitoring Figure 3 is a diagram illustrating the line of sight vector and virtual line of the driver who is the target of monitoring in this embodiment and other embodiments. The symbols explained in Figure 1 will not be explained here. While driving forward, the driver 301 seated in the driver's seat 102 looks out through the windshield, primarily at the front, and its surroundings. However, while driving, the driver does not usually look below, i.e., at the bottom of the lower edge of the windshield, except for a very brief look at meters.

[0022] Therefore, the drive recorder 101 of this embodiment captures the face of the driver 301 using the in-vehicle camera 101R and identifies a line-of-sight vector 302 indicating the direction in which the driver is looking. The line-of-sight vector 302 is a vector that indicates the direction of the driver's gaze, starting from either the left or right eye of the driver. The magnitude of the line-of-sight vector 302 may be an appropriate predetermined value. Whether the left or right eye is the starting point may be determined in advance, or if only one eye can be recognized, the recognized eye may be used as the starting point. If the extension of the line-of-sight vector 302 (shown by a dotted line in the figure) is above an imaginary line 103 indicating the bottom edge of the windshield, it is assumed that the driver is looking outside. However, if the extension of the line-of-sight vector 302 is below the imaginary line 103, it is assumed that the driver is not looking outside and may be looking aside.

[0023] For the above estimation, a three-dimensional coordinate system is set in the drive recorder 101. This is referred to herein as the camera coordinate system. The origin of the camera coordinate system is the installation position of the drive recorder 101, with the X axis set along the optical axis of the interior camera 101R, the Y axis pointing downward toward the vehicle, and the Z axis set in the vehicle width direction. This coordinate system can identify the position of the driver 301 and the position of the virtual line 103. Preferably, the drive recorder 101 is mounted so that the optical axis of the exterior camera 101F is parallel to the surface (ground) on which the vehicle 100 is traveling and aligned with the straight-ahead direction of the vehicle 101. The optical axis of the interior camera 101R is coaxial with or parallel to the optical axis of the exterior camera 101F.

[0024] The mounting position of the drive recorder 101 relative to the vehicle is set when the drive recorder 101 is mounted. The mounting position may include, for example, the height of the drive recorder 101 from the ground and an offset relative to the center line of the vehicle in the width direction. The reason for installing the camera as described above is that the camera coordinate system and the vehicle coordinate system can be easily converted by simply translating them. Therefore, the camera may be installed in any manner as long as it can capture images of the inside and outside of the vehicle. In this case, parameters necessary to maintain compatibility between the camera coordinate system and the vehicle coordinate system are set. Furthermore, in order to estimate the distance to the driver, the distance to the center of the driver's seat relative to the center line of the vehicle in the width direction of the vehicle (the distance in the depth direction in FIG. 3 ) may be set.

[0025] Virtual Line Setting Process Here, the process for setting the virtual line 103 will be described. The process described with reference to FIG. 4 and subsequent figures is executed by the CPU 204, but may also be executed by the image processing unit 203. In either case, the main body of the process may be called the processing unit. The control unit 200 may also be called the processing unit.

[0026] 4, the operator first inputs the position of the bottom edge of the windshield via the input unit 206 (S401). This position may be, for example, the height and depth from the ground. The depth may be, for example, the distance along the X axis from the camera mounting position. If the windshield is a quadratic or cubic surface and its bottom edge is curved, the height and depth may be input based on the front edge. In this case, the virtual line 103 that is set is a straight line that passes through the input position and is parallel to the Z axis.

[0027] Next, the input position is converted into the camera coordinate system (S403). The mounting position of the drive recorder 101 is set and stored when it is installed, so the input position is converted so that this position becomes the origin. Note that when the height along the X axis and the depth along the Y axis are input based on the camera position, they can be converted into the camera coordinates shown in Figure 3 simply by determining the sign of the depth. Note that the length unit is assumed to be a standardized unit such as centimeters or millimeters, which are predetermined units.

[0028] The position coordinates of the virtual line obtained in the final step S403 are written and stored in erasable ROM 209 (S405). The coordinates of the virtual line set here are assumed to be (Xv, Yv). When divided by virtual line (Xv, Yv), if the Y coordinate value of the intersection between the extension of line of sight vector 302 and the Y-Z plane given by the X coordinate value Xv exceeds Yv, it can be estimated that line of sight vector 302 indicates a line of sight looking forward through the front window. On the other hand, if the Y coordinate value of the intersection is Yv or less, it can be estimated that line of sight vector 302 indicates a line of sight looking at dashboard 104 or below, without looking forward.

[0029] Driver Monitoring Process Once the virtual line 103 has been set in the procedure of Fig. 4, the driver is monitored by referring to it. An example of this procedure is shown in Fig. 5. The process of Fig. 5 is a loop that is repeated without ending, but it may be interrupted by an interrupt or the like, and may also be executed in parallel with other processes. The process of Fig. 5 may also be executed by the CPU 204. This process may be started, for example, when the power of the vehicle 100 is turned on and power is supplied to the drive recorder 101. Alternatively, it may be started when the movement of the vehicle is detected from an image captured by the exterior camera 101F.

[0030] First, it is determined whether the driver monitoring setting is on (S501). This setting is stored in the ROM 209 or the like, and if it is on, driver monitoring is performed; if it is not on, it is not performed. In step S501, if the monitoring setting is not on, the process loops at S501, but the process in FIG. 5 may be stopped and execution may start after the monitoring setting is turned on. The driver may be able to set the driver monitoring setting to on or off via the input unit 206, for example.

[0031] If it is determined that the driver monitoring setting is on, it is determined whether the current speed is equal to or greater than a predetermined speed (S503). If it is below the predetermined speed, monitoring is stopped. If the current speed is equal to or greater than the predetermined speed, the process branches to step S505. Note that if the drive recorder 101 does not have a function for measuring speed (or even if it does), step S503 may be skipped. In that case, if the driver monitoring setting is on, the process branches to step S505. For example, the speed can be estimated by the drive recorder 101 as follows. For a target captured in an image captured by the exterior camera 101F, if the contact point between the target and the ground is moved within the screen, the distance from the drive recorder 101 to the contact point can be estimated based on the image height of the contact point in the frame. Therefore, the vehicle speed can be estimated by identifying corresponding stationary targets, such as buildings or white lines on the road, between multiple frames and dividing the difference in distance between the frames of the target by the time difference between the frames.

[0032] Next, the gaze vector 302 of the driver 301 is identified from the image captured by the in-vehicle camera 101R (S505). The gaze vector 302 may be a vector on the X-Y plane as shown in FIG. 3 and may not have a Z component. An existing method may be used to identify the gaze vector. For example, the driver's eyes are recognized from the image. This may be achieved by using pattern matching or a trained model that uses features extracted from the image as input and is machine-learned. Once the eye position is identified, the face direction is identified. The face direction may be identified based on the distortion of an inverted triangle formed by the binoculars and the mouth, using the binocular positions and / or the mouth position as input. The face direction identified in this way may be determined as the gaze vector direction. The magnitude of the gaze vector may be arbitrary. This process may also be performed by machine learning.

[0033] Furthermore, the direction in which the eyes are facing may be identified and combined with the direction of the face to identify the gaze vector. To do this, for example, once the direction of the face has been identified, the image is transformed so that it faces the face directly. In the transformed image, a part near the eye that is not the eyeball (e.g., the inner corner or outer corner of the eye) is identified as the reference position, and the gaze is identified from the positional relationship between the position of the pupil or iris of the eye and the reference position (this will be called a provisional gaze vector). The positional relationship between the position of the pupil or iris of the eye and the reference position and the provisional gaze vector may be associated in advance and saved, and the provisional gaze vector may be identified by referring to this association. Since this provisional gaze vector is a vector when facing the driver's face directly, the provisional gaze vector is transformed so that it is tilted in accordance with the direction of the face, and the desired gaze vector 302 is obtained.

[0034] As described above, the line-of-sight vector 302 starting from the eye can be determined, but because it is a vector based on the eye position, it must be converted into a camera coordinate system. To do this, the eye position, which serves as the starting point, is determined in camera coordinates. Because the drive recorder 101 is fixed relative to the vehicle, if the eyes can be identified from the captured image, the direction of the eyes relative to the in-vehicle camera 101R can be identified. If the in-vehicle camera 101R is a stereo camera, the distance can be determined from the parallax, so the eye position can be identified in polar coordinates with the camera as the origin.

[0035] On the other hand, if the interior camera 101R is a monocular camera, the distance is estimated from the identified eye direction. To this end, for example, it is assumed that the Z-component value of the driver's position (eye position) in the camera coordinate system is determined by the camera's installation position and the seat arrangement in the vehicle. It was explained that, when the drive recorder 101 is installed, for example, the height of the drive recorder 101 from the ground and its offset relative to the vehicle center line are set. In addition, if the interior camera 101R is a monocular camera, an offset in the width direction (Z direction) to the center of the driver's seat 102 relative to the vehicle center line may also be set. This allows the Z-component value (referred to as Zdrv) of the widthwise center line of the driver's seat 102 in the camera coordinate system to be identified. It is assumed that the driver's eye position is located within a plane (referred to as the driver's seat center plane) parallel to the X-Y plane identified by this Z-component value Zdrv. Based on this assumption, if a straight line from the camera toward the driver's eye is identified from the image, the distance from the camera to the intersection of that line and the driver's seat center plane is the distance from the camera to the eye. Once the direction and distance to the eye have been identified, the position in the polar coordinate system can be converted into the camera coordinate system to obtain the starting point of the line-of-sight vector 302. Of course, this method is just one example, and other methods may also be used.

[0036] Next, it is determined whether the gaze vector has been successfully identified (S507). If the driver's face or its direction cannot be identified, it may be determined that the gaze vector has not been identified. If the face direction can be identified but a provisional gaze vector cannot be identified, gaze vector 302 may be identified based on the face direction, and it may be determined that the gaze vector has been successfully identified.

[0037] Once the line of sight vector has been identified, it is determined whether line of sight vector 302 is below the virtual line (S509). That is, it is determined whether an extension of line of sight vector 302 is below virtual line 103. This determination may be made as described above. That is, when a virtual line (Xv, Yv) is set, the intersection of the extension of line of sight vector 302 and the Y-Z plane whose X coordinate value is given by Xv is found. If the Y coordinate value of this intersection is equal to or less than Yv, it can be estimated that line of sight vector 302 is below virtual line 103, that is, that driver 301 is looking at dashboard 104 or below it, rather than looking ahead.

[0038] If it is determined that the extension of line-of-sight vector 302 passes below virtual line 103, it is determined whether a timer has been started (S511), and if not, the timer is started (S513). This timer is, for example, built into CPU 204, and is a timer for measuring a predetermined time in order to determine whether the driver is continuously directing his / her gaze below a preset virtual line for more than a predetermined time.

[0039] On the other hand, if it is determined that the timer has already started, it is then determined whether the predetermined time has expired (S517). The predetermined time may be set when the timer is started, and an interrupt may be generated when the timer expires, or it may be monitored at regular intervals to see if the predetermined time has been reached. In this embodiment, monitoring is performed within a loop process, so the latter method may be used. In particular, the latter method allows the setting of the predetermined time to be changed after the timer has started, and is therefore particularly suitable for the fifth embodiment, which will be described later. This predetermined time may also be called a reference time.

[0040] If the predetermined time has elapsed, the timer is reset and a warning is output from the output unit 207 (S519). This warning may be audible or visual. The warning may be output from the drive recorder 101 itself or from another device such as a mobile terminal. Note that the timer is stopped by being reset.

[0041] If it is determined in step S503 that the speed does not exceed the predetermined speed, or if it is determined in step S507 that the identification of the line-of-sight vector has failed, or if it is determined in step S509 that the extension of the line-of-sight vector is on the virtual line, the process branches to step S515. In step S515, the timer is reset and the output of the warning is stopped. The warning may be stopped even if no warning has been output, or may be stopped only if the warning has been output. Note that in FIG. 5, if the driver monitoring setting is turned off while the warning has been output in step S519, the warning will remain output. In preparation for this case, the process may branch to step S515 if it is determined that the driver monitoring setting is not on.

[0042] Effect of this embodiment In this way, the drive recorder 101 can monitor the driver, particularly whether they are looking ahead. If the driver is looking below the virtual line for more than a predetermined time, it is possible that the driver is staring at the instrument panel, concentrating on operating it, or operating a mobile device. Such situations can be detected by the retrofittable drive recorder, and if there is a possibility, a warning can be issued to the driver, urging them to concentrate on driving.

[0043] [Second Embodiment] In this embodiment, the procedure for setting a virtual line will be described with reference to Fig. 6, which replaces the procedure in Fig. 4. Note that the procedure is the same as that of the first embodiment except that Fig. 6 is replaced by Fig. 4. In this embodiment, instead of an operator directly inputting the position of the virtual line, the driver looks at the virtual line 103, and the position of the virtual line 103 is set from the line-of-sight vector.

[0044] In FIG. 6, first, the line of sight vector of the driver 301 seated in the driver's seat 102 is identified (S601). The line of sight vector may be a vector on the XY plane, and the value in the Z direction does not need to be identified. At this time, the driver is looking at the position where the virtual line 103 between the front and the bottom edge of the window is set. Then, it is determined whether the process is complete (603). "Complete" here means that multiple trials have been completed. In multiple trials, it is desirable that the driver change the height of his face for each trial, for example by changing the depth of his seat. "Multiple times" means at least two trials.

[0045] After the line-of-sight vectors have been identified multiple times, the intersection of the extensions of each line-of-sight vector is identified (S605). The position (X, Y) value of the identified intersection is saved as the setting value of the virtual line (Xv, Yv) (S607).

[0046] Effects of this embodiment As described above, a virtual line can be set without inputting numerical values, etc. This method can also be applied to the third embodiment and the following embodiments.

[0047] [Third Embodiment] Figure 7 shows a driver monitoring procedure in a third embodiment. Figure 7 shows a procedure to be executed in place of step S519 when it is determined in step S517 of Figure 5 that the predetermined time has expired. Therefore, the same steps as in Figure 5 are not described here. In this embodiment, the direction of the face is identified, and the level of warning is changed depending on whether only the eyes are facing down the virtual line or whether the face is also facing down the virtual line.

[0048] 7, if it is determined in step S517 that the predetermined time has elapsed, the direction of the face is identified from the image captured by the in-vehicle camera 101R (S701). This process may be performed as described in step S505 in Fig. 5. However, in step S701, the direction of the face is identified in three dimensions, including the Z component.

[0049] Next, it is determined whether the face is facing downward along the virtual line 103 (S703). For example, a face direction vector is generated instead of the line of sight vector 302, and the intersection of its extension and the Y-Z plane, whose X coordinate value is given by Xv, is found. If the Y coordinate value of the intersection is equal to or less than Yv, it can be estimated that the face is facing downward along the virtual line 103. The Z component of the face direction does not need to be used in this determination.

[0050] If it is determined that the face is not facing down the virtual line 103, it is determined that the vehicle is turning from the images of the exterior camera 101F (S705). To do this, for example, the motion vectors of corresponding objects between multiple frames are detected, and if the average direction is left or right, it may be determined that the vehicle is turning in that direction. The identified turning direction is stored at this time. The turning direction can be expressed, for example, by the sign (positive or negative) of the value obtained by subtracting the Z coordinate value of the end point from the Z coordinate value of the start point of the motion vector. Alternatively, the start point and end point may be reversed.

[0051] Next, it is determined whether the stored turning direction and the face direction match (S707). This determination may be made by checking whether the sign indicating the turning direction stored in step S705 matches the sign of the value obtained by subtracting the Z component of the end point from the Z component of the start point of the vector indicating the face direction. If the signs match, it can be determined that the turning direction and the face direction match.

[0052] If it is determined in step S707 that the two do not match, it is determined that the driver is looking below the virtual line based solely on eye movement, and a normal warning is output (S709). If it is determined that the driver matches, it can be determined that the face is looking in the direction of the turn, so no warning is issued even if the gaze vector is determined to be pointing below the virtual line. Furthermore, if it is determined in step S703 that the face direction is pointing below the virtual line, a stronger warning than normal is output (S711). This is because it is estimated that the driver is distracted. The final timer is reset, and the process branches to step S501 (S713). A stronger warning than normal may be, for example, a loud or intermittent sound in the case of audio, or a more noticeable display color or flashing in the case of a display.

[0053] Effects of this embodiment By following the above procedure, this embodiment can warn the driver more strongly when not only the driver's line of sight but also the driver's face is directed below the virtual line. Furthermore, by not issuing a warning when it is estimated that the driver is looking in the turning direction, it is possible to reduce the annoyance to the driver caused by issuing unnecessary warnings.

[0054] [Fourth Embodiment] Next, an example will be described in which a warning is given to the driver not only when the driver is looking down the windshield, but also when the driver's line of sight is limited to a certain range for a predetermined period of time, i.e., when the driver's line of sight is fixed. Figure 8 shows an example of the procedure executed by the CPU 204 in this case. Figure 8 begins with the description of the case in which it is determined in step S509 of Figure 5 that the extension line of the line of sight vector 302 is not below the virtual line, and the processing in this case is the same as that in the first embodiment. Description of the same parts as in the first embodiment will be omitted.

[0055] In FIG. 8 , if it is determined in step S509 that the extension of the line of sight vector 302 is not below the virtual line, the value of the line of sight vector is saved (S801). The saving location may be RAM 205, with a storage capacity sufficient to save a predetermined number of line of sight vectors. Next, it is determined whether line of sight vectors for a predetermined period of time have been saved (S803). The line of sight vectors for a predetermined period of time are line of sight vectors identified within the predetermined period of time. However, if the amount of data is too large, it is possible to periodically sample and save not all of them but only a portion of them in step S801. Note that in step S801, if the number of saved line of sight vectors exceeds the predetermined period of time, the oldest one may be deleted and a new line of sight vector may be saved.

[0056] If it is determined that gaze vectors for a predetermined period of time have been saved, the system then determines whether the range of the saved gaze vectors, i.e., whether the endpoints of the gaze vectors, fall within a predetermined range (S805). The system may identify the smallest area that includes the endpoints of all saved gaze vectors, and determine that the range of the gaze vectors is limited if its maximum diameter is equal to or less than a threshold. Or, more simply, the system may determine that the range of the gaze vectors is limited if, among all saved gaze vectors, the difference between the maximum and minimum Z components and the difference between the maximum and minimum Y components are both equal to or less than a threshold. In this case, the threshold for the Z component and the threshold for the Y component may be different.

[0057] If it is determined that the end point of the line-of-sight vector is within a predetermined range, the timer is reset and a warning is output (S807), and if not, the timer is reset and the warning is stopped (S809).

[0058] Advantages of this embodiment: By the above procedure, if it is determined that the driver's line of sight is fixed on something, a warning can be output. The predetermined time for collecting line of sight vectors may be a few seconds, for example, 1 to 3 seconds.

[0059] Fifth Embodiment Next, an example will be described in which the warning mode is changed depending on the presence of an obstacle outside the vehicle or a target such as a preceding vehicle, with reference to an image captured by the exterior camera 101F. Figures 9 and 10 show an example of the procedure executed by the CPU 204 in this case. Figure 9 begins with the description of the case in which it is determined in step S511 of Figure 5 that the timer has started, and the process other than that is the same as in the first embodiment. Description of the same parts as in the first embodiment will be omitted.

[0060] If it is determined in step S511 that the timer has started, the target is identified from the image captured by the exterior camera 101F. This identification may be performed by pattern recognition, or by inputting the image features into a trained model that has previously learned the image features and the target in the image through machine learning. As a result, it is determined whether the target has been recognized (S902).

[0061] If it is determined that there are no targets, the process branches to step S501 in FIG. 5. On the other hand, if it is determined that there are targets, the process determines whether the targets include a preceding vehicle (S903). Whether or not a target is a preceding vehicle may be determined, for example, based on its position, size, or distance. If it is determined that there is a preceding vehicle, the process branches to the preceding vehicle processing in FIG. 10. If it is determined that there is no preceding vehicle, the process determines whether there are more than a predetermined number of targets in the image (S905). If it is determined that there are not more than the predetermined number of targets, the normal reference time is set as the expiration time of the timer (S907). The normal reference time may be the time determined as the timer expiration in step S517 in the first embodiment. On the other hand, if it is determined that there are more than the predetermined number of targets, a shortened reference time that is shorter than the normal reference time is set as the expiration time of the timer (S907). Thereafter, the process branches to step S517 to determine whether the timer is full. If the timer is full, step S519 is executed and a warning is output.

[0062] If it is determined in step S903 that there is a leading vehicle, the distance to the leading vehicle is identified in step S911 of FIG. 10 . If the distance was identified when determining the presence of the leading vehicle, that value may be used. In this example, the height of the exterior camera 101F from the ground has already been set, and the direction of its optical axis is assumed to be parallel to the ground and the longitudinal axis of the vehicle. In other words, it is assumed that the vehicle 100 traveling straight travels toward the vanishing point of the image captured by the exterior camera 101F. In this case, the distance to a point set on the ground can be estimated based on the image height of the image. This allows the distance to the leading vehicle to be estimated. Note that even if the installation position and direction of the camera are not as assumed above, the distance to the leading vehicle can be estimated by converting the coordinate system as long as the direction and height of the optical axis relative to the vehicle are known.

[0063] Next, it is determined whether the determined distance exceeds a predetermined distance (S913). If it exceeds the predetermined distance, the process branches to step S907 in Fig. 9 to set the normal reference time, and if it is less than the predetermined distance, the process branches to step S909 to set a shortened reference time. In this way, if the distance to the preceding vehicle is less than the predetermined distance, the shortened predetermined time, i.e., the reference time, is set to the time when the distance exceeds the predetermined distance.

[0064] Advantages of this embodiment: When there are a relatively large number of targets outside the vehicle and when the distance to the leading vehicle is relatively short, the time from when the driver's inattentiveness is detected to when a warning is issued can be shortened compared to other cases. This makes it possible to attract the driver's attention earlier.

[0065] In the above embodiment, the target of monitoring is the driver's behavior of looking below the virtual line. However, this is not limited to this, and the target of monitoring may also be the driver's inattentive behavior. For example, a warning may be output if the driver's line of sight (or face) is directed outside a predetermined range ahead of the vehicle.

[0066] Summary of the embodiment The present embodiment described above can be summarized as follows.

[0067] (1) According to a first aspect of the present invention, there is provided a driver monitoring device that can be retrofitted to a vehicle, comprising: an imaging means for capturing an image including the driver of the vehicle; a control means; and an output means, wherein the control means identifies the driver's line of sight from the image captured by the imaging means; determines whether the identified line of sight is continuously directed below a predetermined virtual line for more than a predetermined time; and outputs a warning to the driver by the output means when it is determined that the line of sight is continuously directed below the virtual line for more than the predetermined time. This makes it possible to grasp a situation in which the driver is looking down and to warn the driver.

[0068] (2) According to a second aspect of the present invention, there is provided the driver monitoring device according to the first aspect, further comprising an input means, wherein the control means sets the virtual line based on the distance and height to the bottom edge of the front windshield of the vehicle relative to a predetermined reference position input by the input means. This makes it possible to determine whether the driver is looking down based on the set virtual line, and therefore to easily and reliably grasp whether the driver is looking down.

[0069] (3) According to a third aspect of the present invention, there is provided the driver monitoring device according to the first aspect, wherein the control means sets the virtual line based on the identified driver's line of sight, thereby enabling the virtual line to be set in a simple manner.

[0070] (4) According to a fourth aspect of the present invention, there is provided the driver monitoring device according to the third aspect, wherein the control means sets the virtual line based on a plurality of different line-of-sight directions, thereby enabling the virtual line to be set in a simple manner.

[0071] (5) According to a fifth aspect of the present invention, there is provided a driver monitoring device according to any one of the first to fourth aspects, wherein the control means identifies the direction in which the driver's face is facing as the gaze direction, thereby enabling the gaze direction of the driver to be identified in a simple manner.

[0072] (6) According to a sixth aspect of the present invention, there is provided a driver monitoring device according to any one of the first to fifth aspects, wherein the control means identifies the direction in which the driver's eyes are facing as the gaze direction. This makes it possible to identify the driver's gaze direction with higher accuracy and to issue a warning to the driver with higher accuracy.

[0073] (7) According to a seventh aspect of the present invention, there is provided a driver monitoring device according to any one of the first to sixth aspects, wherein the control means further identifies the direction of the driver's face, and when it is determined that the driver's line of sight is continuously directed below the virtual line for more than the predetermined time and the identified face direction is directed below the virtual line, the control means outputs the warning in a manner different from when the face direction is not directed below the virtual line. This makes it possible to detect not only the driver's line of sight but also the face direction, and to change the manner of the warning when it is estimated that the driver is distracted.

[0074] (8) According to an eighth aspect of the present invention, there is provided a driver monitoring device according to any one of the first to seventh aspects, wherein the control means further determines whether the identified gaze direction remains within a certain range for a second predetermined time, and outputs a warning from the output means even when it is determined that the gaze direction remains within the certain range for a second predetermined time. This makes it possible to issue a warning not only when the driver is looking down, but also when the driver's gaze is fixed.

[0075] (9) According to a ninth aspect of the present invention, there is provided a driver monitoring device according to any one of the first to eighth aspects, further comprising a second image capturing means for capturing an image of the area ahead of the vehicle, wherein the control means detects targets from the image of the area ahead of the vehicle, and when more than a predetermined number of targets are detected, the control means shortens the predetermined time compared to when no more than the predetermined number of targets are detected. This allows the driver monitoring device to warn the vehicle operator at an earlier timing when there are many obstacles outside the vehicle.

[0076] (10) According to a tenth aspect of the present invention, there is provided the driver monitoring device according to the ninth aspect, wherein the control means, when the detected target is a preceding vehicle, estimates the distance to the preceding vehicle, and when the distance is equal to or less than a predetermined distance, shortens the predetermined time compared to when the distance exceeds the predetermined distance. As a result, when there is a preceding vehicle, a warning can be issued at an earlier timing if the distance between the vehicles is short.

[0077] (11) According to an eleventh aspect of the present invention, there is provided the driver monitoring device according to the ninth aspect, wherein the control means does not cause the output means to output the warning when no target object is detected in the image ahead of the vehicle. This makes it possible to suppress the warning when there is no obstacle in the vicinity.

[0078] (12) According to a twelfth aspect of the present invention, there is provided a driver monitoring device according to any one of the first to eleventh aspects, wherein the control means further identifies the direction of the driver's face, and when the identified face of the driver is directed outside a predetermined range in front of the vehicle, the output means outputs the warning. This makes it possible to issue a warning even when the driver is looking away.

[0079] (13) According to a thirteenth aspect of the present invention, there is provided a driver monitoring device according to the twelfth aspect, further comprising a second image capturing means for capturing an image of the area in front of the vehicle, wherein the control means determines whether the vehicle is turning based on the image of the area in front of the vehicle, and when it is determined that the vehicle is turning and the identified face of the driver matches the direction in which the vehicle is turning, the control means does not output the warning even if the identified face of the driver is facing outside a predetermined range in front of the vehicle. This makes it possible to suppress the warning even if the driver is looking away from the vehicle as long as the direction in which the driver is looking is the turning direction.

[0080] (14) According to a fourteenth aspect of the present invention, there is provided a driver monitoring device according to any one of the first to thirteenth aspects, characterized in that the driver monitoring device functions as a drive recorder that records a certain period of recent video captured by a capturing means. This allows the device to also function as a drive recorder.

[0081] (15) According to a fifteenth aspect of the present invention, there is provided a program for causing a computer to function as the control means according to any one of the first to thirteenth aspects.

[0082] (16) According to a sixteenth aspect of the present invention, there is provided a computer-readable medium storing the program according to the fifteenth aspect.

[0083] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. A driver monitoring device that can be retrofitted to a vehicle, comprising: an imaging means for capturing an image including the vehicle driver; a control means; and an output means, wherein the control means identifies the driver's line of sight from the image captured by the imaging means; determines whether the identified line of sight is continuously directed below a pre-set virtual line for more than a predetermined time; and when it is determined that the line of sight is continuously directed below the virtual line for more than the predetermined time, outputs a warning to the driver via the output means.

2. A driver monitoring device as claimed in claim 1, further comprising an input means, wherein the control means sets the virtual line based on the distance and height to the bottom edge of the vehicle's windshield relative to a predetermined reference position inputted by the input means.

3. A driver monitoring device according to claim 1, characterized in that the control means sets the virtual line based on the identified line of sight of the driver.

4. A driver monitoring device according to claim 3, characterized in that the control means sets the virtual line based on a plurality of different line-of-sight directions.

5. A driver monitoring device according to claim 1, characterized in that said control means specifies the direction in which the driver's face is facing as said line of sight direction.

6. A driver monitoring device according to claim 1, characterized in that the control means identifies the direction in which the driver's eyes are facing as the line of sight direction.

7. A driver monitoring device as described in claim 1, wherein the control means further identifies the direction of the driver's face, and when it is determined that the driver's line of sight is continuously directed below the virtual line for more than the specified time period and the identified direction of the face is directed below the virtual line, outputs the warning in a manner different from when the direction of the face is not directed below the virtual line.

8. A driver monitoring device as described in claim 1, wherein the control means further determines whether the identified gaze direction is within a certain range for a period exceeding a second predetermined time, and outputs a warning via the output means when it is determined that the gaze direction is within the certain range for a period exceeding the second predetermined time.

9. A driver monitoring device as described in claim 1, further comprising a second imaging means for capturing an image of the area in front of the vehicle, wherein the control means detects targets from the image of the area in front of the vehicle, and when more than a predetermined number of targets are detected, the control means shortens the predetermined time compared to when no more than the predetermined number of targets are detected.

10. A driver monitoring device as described in claim 9, wherein the control means estimates the distance to the preceding vehicle when the detected target object is a preceding vehicle, and when the distance is equal to or less than a predetermined distance, shortens the predetermined time compared to when the distance exceeds the predetermined distance.

11. A driver monitoring device as claimed in claim 9, characterized in that the control means does not cause the output means to output the warning when no target object is detected in an image ahead of the vehicle.

12. A driver monitoring device as described in claim 1, wherein the control means further identifies the direction of the driver's face, and when the identified face of the driver is directed outside a predetermined range in front of the vehicle, outputs the warning by the output means.

13. A driver monitoring device as described in claim 12, further comprising a second imaging means for capturing an image of the front of the vehicle, wherein the control means determines whether the vehicle is turning based on the image of the front of the vehicle, and when it is determined that the vehicle is turning and the identified face of the driver matches the direction in which the vehicle is turning, the control means does not output the warning even if the identified face of the driver is facing outside a specified range in front of the vehicle.

14. A driver monitoring device according to any one of claims 1 to 13, characterized in that the driver monitoring device functions as a drive recorder that records a certain period of the most recent video captured by the imaging means.

15. A program for causing a computer to function as the control means according to any one of claims 1 to 13.

16. A computer readable medium storing the program according to claim 15.