Determination device, computer program product for determination, and determination method

Abstract

The invention relates to a determination device, a computer program product for determination, and a determination method. The present disclosure provides a determination device capable of determining that a driver is dozing without erroneously detecting the smiling of the driver. The determination device is provided with a determination unit that determines that the degree of eye opening of the driver is equal to or less than a reference degree of eye opening, and that the orientation of the face of the driver indicates a movement exceeding a first reference in the right direction, and determines that the degree of eye opening of the driver is equal to or less than the reference degree of eye opening. The driver is determined not to be drowsy when the driver is determined to have an eye-opening degree equal to or less than a reference eye-opening degree, and the driver is determined not to have drowsy when the driver's face orientation is determined to have a movement exceeding a second reference in the left direction in succession with the movement. If it is determined that the driver does not have a movement in the right direction exceeding the first reference and that the driver does not have a movement in the left direction exceeding the second reference in succession with the movement, it is determined that the driver is dozing.

Description

Technical Field

[0001] The present disclosure relates to a determination device, a determination computer program product, and a determination method. Background Art

[0002] Conventionally, a monitoring device mounted on a vehicle monitors the driver's condition. For example, the monitoring device monitors whether the driver is dozing off.

[0003] The monitoring device checks the driver's eye openness based on an image of their face to determine whether they are dozing off. If a low eye openness (closed eyes) is detected for a predetermined period of time, the monitoring device determines that the driver is dozing off and issues a warning to the driver.

[0004] For example, Patent Document 1 proposes a technology for determining driver drowsiness based on eyelid movement and changes in facial orientation angle immediately following eyelid movement. Patent Document 1 states that if the facial orientation angle immediately after closing the eyelids moves to the left or right by more than a specified amount, the driver is determined to be not drowsy.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2011-048531

[0008] However, a driver may sometimes fall asleep with his face turned in one direction, and therefore, the technology proposed in Patent Document 1 may not be able to detect the driver's drowsiness.

[0009] Furthermore, when the driver is smiling, the degree of eye opening may be less than or equal to the threshold. Detecting only when the facial orientation angle has shifted to the left or right by a predetermined amount or more immediately after the eyelids are closed may fail to accurately detect that the driver is smiling. Summary of the Invention

[0010] Therefore, the present disclosure sets the following as an object: to provide a determination device that can determine whether a driver is dozing off without falsely detecting a driver's smile.

[0011] (1) According to one embodiment, there is provided a determination device. The determination device has: a first determination section that determines whether or not an eye opening degree of a driver is equal to or less than a reference eye opening degree; a second determination section that determines, based on information that represents movement of a face of the driver, whether or not the face of the driver indicates movement in a right direction that exceeds a first reference, and, continuously with the movement, the face of the driver indicates movement in a left direction that exceeds a second reference; and a third determination section that determines that the driver is not dozing off, in a case where the first determination section determines that the eye opening degree of the driver is equal to or less than the reference eye opening degree, the second determination section determines that the face of the driver indicates movement in the right direction that exceeds the first reference, and, continuously with the movement, the face of the driver indicates movement in the left direction that exceeds the second reference, and determines that the driver is dozing off, in a case where the first determination section determines that the eye opening degree of the driver is equal to or less than the reference eye opening degree, and the second determination section determines that the face of the driver does not indicate movement in the right direction that exceeds the first reference, and, continuously with the movement, the face of the driver does not indicate movement in the left direction that exceeds the second reference.

[0012] (2) In the determination device of (1), it is preferable that the information that represents movement of the face of the driver is represented by an angle of the face of the driver in a horizontal direction, an angular velocity of the face of the driver in the horizontal direction, or an angular acceleration of the face of the driver in the horizontal direction.

[0013] (3) In the determination device of (1) or (2), it is preferable that, in a case where the second determination section determines that the face of the driver indicates movement in the right direction that exceeds the first reference, and, continuously with the movement, the face of the driver indicates movement in the left direction that exceeds the second reference, within a reference time from a time point at which it is determined that the driver is not dozing off, the third determination section determines that the driver is not dozing off.

[0014] (4) In any one of the determination devices of (1) to (3), it is preferable to further have a fourth determination section that determines whether the orientation of the driver's face indicates movement in the upward direction exceeding a third reference or the orientation of the driver's face indicates movement in the downward direction exceeding a fourth reference, based on the movement of the driver's face, and in the case where it is determined by the first determination section that the degree of opening of the driver's eyes is less than or equal to the reference degree of opening and it is determined by the fourth determination section that the orientation of the driver's face indicates movement in the upward direction exceeding the third reference or the orientation of the driver's face indicates movement in the downward direction exceeding the fourth reference, even in the case where it is determined by the second determination section that the orientation of the driver's face indicates movement in the rightward direction exceeding the first reference and, continuously with this movement, the orientation of the driver's face indicates movement in the leftward direction exceeding the second reference, the third determination section determines that the driver is dozing off.

[0015] (5) According to another embodiment, a determination computer program product is provided. The determination computer program product causes a processor to perform: determining whether a degree of opening of a driver's eyes is less than or equal to a reference degree of opening; determining, based on information that represents movement of the driver's face, whether the orientation of the driver's face indicates movement in the rightward direction exceeding a first reference and, continuously with this movement, the orientation of the driver's face indicates movement in the leftward direction exceeding a second reference; and in the case where it is determined that the degree of opening of the driver's eyes is less than or equal to the reference degree of opening and it is determined that the orientation of the driver's face indicates movement in the rightward direction exceeding the first reference and, continuously with this movement, the orientation of the driver's face indicates movement in the leftward direction exceeding the second reference, determining that the driver is not dozing off, and, in the case where it is determined that the degree of opening of the driver's eyes is less than or equal to the reference degree of opening and it is determined that the orientation of the driver's face does not indicate movement in the rightward direction exceeding the first reference and, continuously with this movement, the orientation of the driver's face does not indicate movement in the leftward direction exceeding the second reference, determining that the driver is dozing off.

[0016] (6) According to still another embodiment, there is provided a determination method. The determination method includes determining, by a determination device, whether or not an eye opening degree of a driver is equal to or less than a reference eye opening degree; determining, based on information representing movement of a face of the driver, whether or not the face of the driver indicates movement in a right direction exceeding a first reference, and, continuously with the movement, the face of the driver indicates movement in a left direction exceeding a second reference; and determining that the driver is not dozing off when it is determined that the eye opening degree of the driver is equal to or less than the reference eye opening degree, and it is determined that the face of the driver indicates movement in the right direction exceeding the first reference, and, continuously with the movement, the face of the driver indicates movement in the left direction exceeding the second reference, and determining that the driver is dozing off when it is determined that the eye opening degree of the driver is equal to or less than the reference eye opening degree, and it is determined that the face of the driver does not indicate movement in the right direction exceeding the first reference, and, continuously with the movement, the face of the driver does not indicate movement in the left direction exceeding the second reference.

[0017] Effects of Invention

[0018] The determination device of the present disclosure can determine that the driver is dozing off without erroneously detecting a smile of the driver's face. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a diagram that outlines the operation of the monitoring device of the first embodiment.

[0020] Figure 2 is a diagram that outlines the operation of the monitoring device of the first embodiment.

[0021] Figure 3 is a diagram that outlines the operation of the monitoring device of the first embodiment.

[0022] Figure 4 is a diagram that outlines the operation of the monitoring device of the first embodiment.

[0023] Figure 5 is a diagram that outlines the operation of the monitoring device of the first embodiment.

[0024] Figure 6 is a diagram that outlines the operation of the monitoring device of the first embodiment.

[0025] Figure 7 is a diagram that outlines the operation of the monitoring device of the first embodiment. DETAILED DESCRIPTION

[0026] Figure 1is a diagram that explains an outline of the operation of the monitoring device of the first embodiment. As shown in Figure 1 The vehicle 10 has the monitoring device 11. Inside the cabin 10A of the vehicle 10, the driver 30 is seated on the driver's seat 40. The monitoring camera 2 is disposed on the dashboard.

[0027] The monitoring device 11 monitors the driver 30 on the basis of a monitoring image acquired by the monitoring camera 2. The monitoring device 11 is one example of a determination device. The vehicle 10 can also be an automated vehicle.

[0028] The monitoring camera 2 can acquire an image that represents the face of the driver 30. The monitoring device 11 detects the eye opening degree of the driver 30 on the basis of the monitoring image acquired by the monitoring camera 2. In addition, the monitoring device 11 detects information that represents the movement of the face of the driver 30 on the basis of the monitoring image.

[0029] The monitoring device 11 determines whether or not the driver 30 is dozing off on the basis of the eye opening degree of the driver 30 and the information that represents the movement of the face of the driver 30. In a case where it is determined that the driver 30 is dozing off, the monitoring device 11 notifies the driver 30 of a warning via a user interface (UI: User Interface) 4.

[0030] The monitoring device 11 determines whether or not the driver 30 is in a closed-eye state on the basis of the eye opening degree of the driver 30. Here, in a case where the driver 30 is smiling, a state in which the eye opening degree is low is sometimes sensed for a prescribed time. In order to prevent a false determination that the driver is dozing off, it is desirable to be able to correctly determine whether or not the driver is smiling or dozing off.

[0031] As a result of investigation by the inventor of the present disclosure, it was found that a person makes the orientation of the face continuously change to the left and right when smiling. A person sometimes makes the face orient in one direction in a manner that makes the head stable when asleep, but does not make the orientation of the face continuously change to the left and right when asleep. Making the orientation of the face continuously change to the left and right is a characteristic of a person when smiling.

[0032] Therefore, in a case where it is determined that the driver 30 is in a closed-eye state, the monitoring device 11 determines whether or not the orientation of the face is being made to continuously change to the left and right on the basis of the information that represents the movement of the face of the driver 30.

[0033] In a case where it is determined that the eye openness degree of the driver 30 is equal to or smaller than the reference eye openness degree, the monitoring device 11 investigates the movement of the face of the driver 30. In a case where it is determined that the orientation of the face of the driver 30 indicates movement in the right direction exceeding the first reference, and continuously with this movement, the orientation of the face of the driver 30 indicates movement in the left direction exceeding the second reference, the monitoring device 11 determines that the driver 30 is not dozing off.

[0034] Even if it is determined that the driver 30 is closing the eyes, in a case where the orientation of the face is being continuously changed in the right and left directions, the driver 30 is in a state of being in laughter. By determining whether or not the orientation of the face is being continuously changed in the right and left directions, it is possible to prevent the laughter of the driver 30 from being erroneously determined as being dozing off.

[0035] Further, in a case where it is determined that the eye openness degree of the driver 30 is equal to or smaller than the reference eye openness degree, and it is determined that the orientation of the face of the driver 30 does not indicate movement in the right direction exceeding the first reference, and continuously with this movement, the orientation of the face of the driver 30 does not indicate movement in the left direction exceeding the second reference, the monitoring device 11 determines that the driver 30 is dozing off.

[0036] That is, in a case where it is determined that the driver 30 is closing the eyes and the orientation of the face is not being continuously changed in the right and left directions, it is presumed that the driver 30 is dozing off.

[0037] In a case where it is determined that the driver 30 is dozing off, the monitoring device 11 notifies the driver 30 of a warning via the UI 4, thereby increasing the degree of participation in driving.

[0038] As explained above, the monitoring device 11 of the present embodiment can determine dozing off in a manner that does not erroneously detect the laughter of the driver 30.

[0039] Figure 2 is a schematic configuration view of the vehicle 10 on which the monitoring device 11 of the present embodiment is installed. The vehicle 10 has a monitoring camera 2, a user interface (UI) 4, and a monitoring device 11, and the like.

[0040] The monitoring camera 2, the UI 4, and the monitoring device 11 are communicably connected via an in-vehicle network 12 conforming to a standard such as a controller area network.

[0041] The monitoring camera 2 is disposed in the vehicle cabin 10A in a manner that can acquire a monitoring image including the face of the driver 30 who is driving the vehicle 10. The monitoring camera 2 is one example of an image acquisition unit. For example, the monitoring camera 2 is disposed so as to be able to acquire an image of the vicinity of the driver's seat 40. As shown in Figure 1 for example, above the instrument panel.

[0042] The monitoring camera 2 acquires a monitoring image at an image acquisition timing with a prescribed period, for example. The monitoring camera 2 outputs the monitoring image and the image acquisition timing to the monitoring device 11 via the in-vehicle network 12 every time a monitoring image is acquired. The prescribed period can be set to 0.1 to 0.5 seconds, for example.

[0043] The monitoring camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to infrared rays, such as a CCD (Charge Coupled Device) or a C-MOS (Complementary Metal Oxide Semiconductor), and a photographing optical system that images an image of a region that becomes a photographic subject onto the two-dimensional detector. Preferably, the monitoring camera 2 has the two-dimensional detector, and has a light projector. The light projector is, for example, two near-infrared LEDs (Light Emitting Diodes) disposed on both sides of the photographing optical system. By irradiating the driver 30 with near-infrared light, the driver 30 can be photographed without causing the driver 30 to feel uncomfortable even in low illumination such as at night.

[0044] The UI 4 is an example of a notification section. The UI 4 is controlled by the monitoring device 11, and notifies the driver 30 of a warning or the like. The UI 4 has a display device 4a, such as a liquid crystal display or a touch panel, to display a warning or the like. In addition, the UI 4 can also have a sound output device (not shown) to notify the driver 30 of a warning or the like. In addition, the UI 4 has, for example, a touch panel or an operation button as an input device to input operation information from the driver 30 to the vehicle 10. The UI 4 outputs the input operation information to the monitoring device 11 via the in-vehicle network 12.

[0045] The monitoring device 11 performs a detection process, a determination process, and a control process. To this end, the monitoring device 11 has a communication interface (I / F) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit to connect the monitoring device 11 to the in-vehicle network 12.

[0046] The memory 22 is an example of a storage section, and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. In addition, the memory 22 stores a computer program product of an application program used in information processing performed by the processor 23 and various data. The monitoring image input from the monitoring camera 2 is stored in association with the image acquisition timing in the memory 22.

[0047] All or a part of the functions of the monitoring device 11 are implemented by functional modules of a computer program that operates on the processor 23, for example. The processor 23 has a detection section 231, a determination section 232, and a control section 233. The determination section 232 is an example of the first determination section, the second determination section, and the third determination section. The functional modules of the processor 23 can also be dedicated arithmetic circuits provided in the processor 23.

[0048] The processor 23 has one or a plurality of CPUs (Central Processing Units) and peripheral circuits thereof. The processor 23 can further have other arithmetic circuits such as a logical operation unit, a numerical operation unit, or a graphics processing unit. The monitoring device 11 is an electronic control unit (ECU), for example.

[0049] The detection section 231 obtains the eye openness of the driver 30 based on the plurality of monitoring images at the eye openness detection timing that has a prescribed period. The period can be set to 0.1 seconds to 5 seconds, for example.

[0050] The detection section 231 has a classifier that is learned to detect the region of the eye with the input monitoring image. In the classifier, the monitoring image is input to detect the region of the eye included in the monitoring image.

[0051] The classifier is a deep neural network (DNN) that has a plurality of layers connected in series from the input side toward the output side, for example. The DNN is learned by inputting images including eyes as training data in advance, and thereby functions as a classifier that detects the region of the eye. In addition, a machine learning model such as a support vector machine or a random forest can be used as the classifier.

[0052] In addition, the detection section 231 obtains the distance from the monitoring camera 2 to the face of the driver 30. For example, the detection section 231 obtains the distance from the monitoring camera 2 to the face of the driver 30 based on the size of the face of the driver 30 represented in the monitoring image with reference to the relationship between the size of the face of a standard person represented in the image and the distance. The region of the face of the driver 30 in the monitoring image is detected using a classifier that is learned to detect the region of the face.

[0053] The detection section 231 obtains the eye openness based on the distance from the monitoring camera 2 to the face of the driver 30 and the number of pixels between the upper eyelid and the lower eyelid in the monitoring image. For example, the detection section 231 can obtain the maximum number of pixels between the upper eyelid and the lower eyelid. The eye openness is represented in mm, for example. The detection section 231 notifies the determination section 232 of the eye openness of each of the right eye and the left eye of the driver.

[0054] At the information detection timing with a prescribed cycle, the detection section 231 detects information representing the movement of the face of the driver 30 based on the monitoring image. The cycle can be set to 0.1 to 1 seconds, for example. The detection section 231 notifies the determination section 232 of the information representing the movement of the face of the driver 30.

[0055] The information representing the movement of the face of the driver 30 is represented by, for example, the angle of the orientation of the face of the driver 30 in the horizontal direction, the angular velocity of the orientation of the face of the driver 30 in the horizontal direction, or the angular acceleration of the orientation of the face of the driver 30 in the horizontal direction.

[0056] The orientation of the face of the driver 30 is represented by, for example, the angle in the horizontal direction between the direction of travel of the vehicle 10 and the direction in which the driver 30 has the face oriented. For example, the direction of travel of the vehicle 10 is set to 0 degrees, the orientation of the face of the driver 30 facing to the left is represented by an angle of 0 to -180 degrees, and the orientation of the face of the driver 30 facing to the right is represented by an angle of 0 to 180 degrees.

[0057] The detection section 231 has a classifier that has been learned to detect prescribed parts of the face such as the outer canthus, the inner canthus, and the corner of the mouth from an image. The detection section 231 determines the positions of the prescribed parts included in the monitoring image by inputting the monitoring image to the classifier. Then, the detection section 231 collates the positions of the prescribed parts detected from the monitoring image with a standard three-dimensional model of the face. Then, the detection section 231 determines the angle of the orientation of the face in the three-dimensional model at which the positions of the respective parts most match the positions of the parts detected from the monitoring image as the angle of the orientation of the face in the monitoring image.

[0058] The classifier can employ, for example, a deep neural network (DNN) having a plurality of layers connected in series from the input side toward the output side. A face image including the prescribed parts of the face is used as training data in advance and is input to the DNN to be learned, whereby the DNN functions as a classifier that determines the positions of the prescribed parts of the face.

[0059] The detection section 231 obtains the time-series change in the angle of the orientation of the face of the driver 30 in the horizontal direction. Note that the detection section 231 can also calculate the orientation of the face of the driver 30 based on the direction of the line of sight of the driver 30.

[0060] Further, the detection section 231 can also calculate the angular velocity of the orientation of the face of the driver 30 in the horizontal direction as the amount of change per unit time of the angle of the orientation of the face of the driver 30 in the horizontal direction. Thus, the detection section 231 obtains the time-series change in the angular velocity of the orientation of the face of the driver 30 in the horizontal direction.

[0061] Moreover, the detection section 231 can also obtain the horizontal-directional angular acceleration of the orientation of the face of the driver 30 as the amount of change per unit time of the horizontal-directional angular velocity of the orientation of the face of the driver 30. Thereby, the detection section 231 obtains the time-series change of the horizontal-directional angular acceleration of the orientation of the face of the driver 30.

[0062] Furthermore, the detection section 231 can also detect the vertical-directional movement of the face of the driver 30. The vertical-directional movement of the orientation of the face of the driver 30 is expressed by, for example, the vertical-directional angle between the traveling direction of the vehicle 10 and the direction in which the driver 30 is orienting the face. Moreover, the detection section 231 can also obtain the vertical-directional angular velocity or angular acceleration of the orientation of the face of the driver 30.

[0063] For example, the traveling direction of the vehicle 10 is set to 0 degrees, the orientation of the face of the driver 30 upward is expressed by an angle of 0 degrees to 90 degrees, and the orientation of the face of the driver 30 downward is expressed by an angle of 0 degrees to -90 degrees.

[0064] Figure 3 is one example of an action flowchart related to the monitoring processing of the monitoring device 11 of the present embodiment. Next, hereinafter, the monitoring processing of the monitoring device 11 will be described with reference to Figure 3 The monitoring processing of the monitoring device 11 will be described. At the monitoring time having a prescribed period, the monitoring device 11 executes the monitoring processing in accordance with the action flowchart shown in Figure 3

[0065] First, the determination section 232 obtains the average value of the eye opening degree of the driver 30 (step S101). For example, the determination section 232 obtains the average value of the eye opening degree obtained within a certain period immediately before, for each of the left and right eyes of the driver 30. As the certain period immediately before, 10 seconds to 30 seconds can be set.

[0066] Next, the determination section 232 determines whether or not the driver 30 is in the closed-eye state on the basis of the eye opening degree of the driver 30 (step S102). For example, in a case where the average value of the eye opening degree of the left and right eyes of the driver 30 is each less than or equal to a reference eye opening degree, the determination section 232 determines that the driver 30 is in the closed-eye state. As the reference eye opening degree, for example, 3 mm can be set. Generally, the eye opening degree of a person is in the range of 10 mm to 30 mm.

[0067] On the other hand, in a case where at least one of the average values of the eye opening degree of the left and right eyes of the driver 30 is not less than or equal to the reference eye opening degree, the determination section 232 determines that the driver 30 is not in the closed-eye state.

[0068] ​In a case where it is determined that the driver 30 is in the closed-eye state (step S102 - Yes), the determination section 232 determines whether the orientation of the face of the driver 30 indicates movement in the right direction exceeding a first reference, based on the information representing the movement of the face of the driver 30 during a determination period (step S103). For example, the determination period can be set to 1 to 5 seconds. The determination period can start from the time when it is determined that the driver 30 is in the closed-eye state. Further, it can be set that the determination period includes the time when it is determined that the driver 30 is in the closed-eye state.

[0069] In a case where the information representing the movement of the face of the driver 30 is represented in an angle, the first reference can be set to +10 degrees, for example. In a case where the information representing the movement of the face of the driver 30 is represented in an angular velocity, the first reference can be set to +2.5 degrees / second, for example. In a case where the information representing the movement of the face of the driver 30 is represented in an angular acceleration, the first reference can be set to +2.5 degrees / second 2 .

[0070] The determination section 232 investigates the information representing the movement of the face of the driver 30 from the start time of the determination period. In a case where a value of the information representing the movement of the face of the driver 30 exceeding the first reference exists, the determination section 232 determines that the orientation of the face of the driver 30 indicates movement in the right direction exceeding the first reference.

[0071] In a case where the orientation of the face of the driver 30 indicates movement in the right direction exceeding the first reference (hereinafter, also referred to as a first action) (step S103 - Yes), the determination section 232 determines whether the orientation of the face of the driver 30 indicates movement in the left direction exceeding a second reference, continuously with the first action, based on the information representing the movement of the face of the driver 30 during a determination period (step S104). The determination period is the same period as the determination of the first action. The information representing the movement of the face of the driver 30 uses the same information as the first action.

[0072] The determination section 232 investigates the information representing the movement of the face of the driver 30 from the start time of the determination period. In a case where a value of the information representing the movement of the face of the driver 30 exceeding the second reference exists continuously with the first action, the determination section 232 determines that the orientation of the face of the driver 30 indicates movement in the left direction exceeding the second reference (hereinafter, also referred to as a second action). The second action sometimes occurs before the first action, and sometimes occurs after the first action

[0073] In a case where the information representing the movement of the face of the driver 30 is represented in an angle, the second reference can be set to, for example, -10 degrees. In a case where the information representing the movement of the face of the driver 30 is represented in an angular velocity, the second reference can be set to, for example, -2.5 degrees / second. In a case where the information representing the movement of the face of the driver 30 is represented in an angular acceleration, the second reference can be set to, for example, -2.5 degrees / second 2 The absolute value of the second reference can be the same as the first reference or different from the first reference.

[0074] In a case where it is determined that the orientation of the face of the driver 30 represents a movement in the left direction exceeding the second reference (step S104 - Yes) or in a case where it is determined that the driver 30 is not in the closed-eye state (step S102 - No), the determination section 232 determines that the driver 30 is not dozing off (step S105), and ends the series of processes.

[0075] On the other hand, in a case where the orientation of the face of the driver 30 does not represent a movement in the right direction exceeding the first reference (step S103 - No) or in a case where the orientation of the face of the driver 30 does not represent a movement in the left direction exceeding the second reference (step S104 - No), the determination section 232 determines that the driver 30 is dozing off (step S106).

[0076] Note that, further determination processing of whether or not in the closed-eye state can be performed between step S103 and step S106 and between step S104 and step S106. It can be provided that, after determining that the driver 30 is in the closed-eye state during a prescribed time (for example, 3 to 10 seconds), the determination section 232 proceeds to step S106. In a case where it is determined that the driver 30 is not in the closed-eye state during the period, the series of processes ends.

[0077] The fact that the orientation of the face of the driver 30 represents a movement in the right direction exceeding the first reference and, continuously with the movement, the orientation of the face of the driver 30 represents a movement in the left direction exceeding the second reference includes: the absence of the first movement and the second movement; the presence of the first movement but the absence of the second movement; and the absence of the first movement but the presence of the second movement.

[0078] Next, the control section 233 notifies the driver 30 of a warning via the UI 4 (step S107), and ends the series of processes. For example, the control section 233 causes the display device 4a to display a warning and output a large warning sound via the UI 4. Thereby, the monitoring device 11 increases the degree of participation of the driver 30 in driving.

[0079] Figure 4is a graph indicating a change in the orientation of the face representing the movement of the face of the driver 30. The vertical axis indicates the angle in the horizontal direction of the orientation of the face, and the horizontal axis indicates time. At time zero, it is determined that the driver 30 is in the closed-eye state. During the period Tm from time zero, the determination section 232 determines the movement of the face of the driver 30.

[0080] The angle representing the orientation of the face exceeds the first reference + Thl at time Tl, and exceeds the second reference - Thl at time T2. The second action in which the angle exceeds the second reference - Thl at time T2 is continuous with the first action in which the angle exceeds the first reference + Thl at time Tl. Time T2 is within the period Tm. It is presumed that the driver 30 is smiling. Therefore, the determination section 232 determines that the driver 30 is not dozing off.

[0081] Figure 5 is a graph indicating a change in the angular velocity of the movement of the face representing the movement of the face of the driver 30. The vertical axis indicates the angular velocity in the horizontal direction of the orientation of the face, and the horizontal axis indicates time. At time zero, it is determined that the driver 30 is in the closed-eye state. During the period Tm from time zero, the determination section 232 determines the movement of the face of the driver 30.

[0082] The angular velocity representing the movement of the face exceeds the first reference + Th2 at time Tl, and exceeds the second reference - Th2 at time T2. The second action in which the angular velocity exceeds the second reference - Th2 at time T2 is continuous with the first action in which the angular velocity exceeds the first reference + Th2 at time Tl. Time T2 is within the period Tm. It is presumed that the driver 30 is smiling. Therefore, the determination section 232 determines that the driver 30 is not dozing off.

[0083] In order to represent the above-described angle representing the orientation of the face, it is necessary to set the traveling direction of the vehicle 10 as the reference of the angle. On the other hand, the angular velocity is represented by the amount of change in the angle per unit time, and thus it is not necessary to set the reference, and thus the use of the angular velocity makes the processing easy. In the case where the movement of the face of the driver 30 is represented by angular acceleration, the processing is also made easy.

[0084] According to the monitoring device of the above-described embodiment, it is possible to determine dozing off without erroneously detecting a smile of the driver.

[0085] Next, the following will be described with reference to Figure 6 and Figure 7 The monitoring device of the second and third embodiments disclosed in this specification will be described. With regard to the monitoring device of the present embodiment, the description of the monitoring device of the first embodiment described above is appropriately applied to points for which no particular description is made.

[0086] Figure 6is one example of an action flowchart related to the monitoring processing of the monitoring device 11 of the second embodiment. In the present embodiment, the processing of steps S206 to S208 is added, which is different from the first embodiment described above. The processing of steps S201 to S205, step S209, and step S210 is the same as steps S101 to S107 described above.

[0087] In the present embodiment, after it is determined that the driver 30 is not dozing off (step S205), the determination section 232 determines whether the orientation of the face of the driver 30 indicates movement in the right direction exceeding the first reference (step S206).

[0088] The determination section 232 determines whether the orientation of the face of the driver 30 indicates movement in the right direction exceeding the first reference, based on information representing the movement of the face of the driver 30 during the determination period. For example, the determination period can be set to 1 to 5 seconds. The determination period can start from the time when it is determined that the driver 30 is not dozing off.

[0089] In the case where the orientation of the face of the driver 30 indicates movement in the right direction exceeding the first reference (step S206 - Yes), the determination section 232 determines whether the orientation of the face of the driver 30 indicates movement in the left direction exceeding the second reference continuously from the first action (step S207), based on information representing the movement of the face of the driver 30 during the determination period.

[0090] In the case where the orientation of the face of the driver 30 indicates movement in the left direction exceeding the second reference (step S207 - Yes), the determination section 232 determines whether the time from the point when it is determined that the driver 30 is not dozing off is within the reference time (step S208). The determination section 232 determines whether the time when the second action is confirmed is within the reference time from the point when it is determined that the driver 30 is not dozing off. As the reference time, for example, 3 to 10 seconds can be set.

[0091] In the case where it is within the reference time (step S208 - Yes), the determination section 232 determines that the driver 30 is not dozing off (step S205). Thereafter, the processing proceeds to step S206.

[0092] On the other hand, in the case where the orientation of the face of the driver 30 does not indicate movement in the right direction exceeding the first reference (step S206 - No), in the case where the orientation of the face of the driver 30 does not indicate movement in the left direction exceeding the second reference (step S207 - No), or in the case where it is not within the reference time (step S208 - No), the series of processing ends. It is presumed that the driver 30 is not laughing, and thus the monitoring processing starts again from the monitoring of the closed-eye state.

[0093] Below, refer to Figure 4 and Figure 5 The monitoring process of this embodiment is described. Figure 4 In the example shown, first, at time T2, it is determined that the driver 30 is not dozing off. Therefore, the determination unit 232 does not determine the eye-closing state, but continues to determine the movement of the driver 30's face.

[0094] The angle representing the facial orientation exceeds the first reference + Th1 at time T3 and then exceeds the second reference - Th1 at time T4. The second movement in which the angle exceeds the second reference - Th1 at time T4 is continuous with the first movement in which the angle exceeds the first reference + Th1 at time T3.

[0095] Time T4 is within the reference time Tn from time T2 at which it is determined that the driver 30 is not dozing off. Therefore, the determination unit 232 determines that the driver 30 is not dozing off.

[0096] In addition, Figure 5 In the example shown, first, at time T2, it is determined that the driver 30 is not dozing off. Therefore, the determination unit 232 does not determine the eye-closing state, but continues to determine the movement of the driver 30's face.

[0097] The angular velocity representing the movement of the face exceeds the first reference + Th2 at time T3 and then exceeds the second reference - Th2 at time T4. The second movement in which the angular velocity exceeds the second reference - Th2 at time T4 is continuous with the first movement in which the angular velocity exceeds the first reference + Th2 at time T3.

[0098] Time T4 is within the reference time Tn from time T2 at which it is determined that the driver 30 is not dozing off. Therefore, the determination unit 232 determines that the driver 30 is not dozing off.

[0099] In the present embodiment, when it is determined that the driver 30 is not dozing off, it is estimated that the driver 30 is smiling. Therefore, in the present embodiment, the eye-closing state is not determined, and the determination of the movement of the driver's 30 face is continued.

[0100] As described above in detail, the monitoring device of this embodiment can reduce the processing load of the monitoring device. In addition, the monitoring device of this embodiment has the same effects as the first embodiment described above.

[0101] Figure 7is one example of an action flowchart related to the monitoring processing of the monitoring device 11 of the third embodiment. In the present embodiment, the processing of step S305 is added, which is different from the first embodiment described above. The processing of steps S301 to S304 and steps S306 to S308 is the same as steps S101 to S107 described above.

[0102] In the case where the orientation of the face of the driver 30 indicates movement in the left direction exceeding the second reference (step S304 - Yes), the determination section 232 determines, based on the movement of the face of the driver 30, whether the orientation of the face of the driver 30 indicates movement in the upward direction exceeding a third reference or indicates movement in the downward direction exceeding a fourth reference (step S305).

[0103] The determination section 232 acquires, from the detection section 231, information representing movement in the vertical direction of the face of the driver 30. The information representing movement in the vertical direction of the face of the driver 30 is represented in terms of angle, angular velocity, or angular acceleration.

[0104] The determination section 232 determines, based on the information representing movement in the vertical direction of the face of the driver 30, whether the orientation of the face of the driver 30 indicates movement in the upward direction exceeding the third reference or indicates movement in the downward direction exceeding the fourth reference. The description of the processing of determining the information representing movement in the horizontal direction of the face of the driver 30 described above is appropriately applied to the determination of the information representing movement in the vertical direction of the face of the driver 30. For example, as the third reference, 10 degrees can be set, and as the fourth reference, -10 degrees can be set.

[0105] In the case where the orientation of the face of the driver 30 indicates movement in the upward direction exceeding the third reference or indicates movement in the downward direction exceeding the fourth reference (step S305 - Yes), the determination section 232 determines that the driver 30 is dozing off (step S307).

[0106] In the case where the orientation of the face of the driver 30 indicates movement in the upward direction or the downward direction exceeding the reference, the possibility that the driver 30 is dozing off is high, and thus it is determined that the driver 30 is dozing off regardless of the movement in the horizontal direction of the face of the driver 30.

[0107] As described in detail above, according to the monitoring device of the present embodiment, it is possible to reliably determine that the driver 30 is dozing off. Furthermore, according to the monitoring device of the present embodiment, the same effects as the first embodiment described above are exerted.

[0108] In the present disclosure, the determination device, the determination computer program, and the determination method of the above-described embodiments can be appropriately changed as long as the gist of the present disclosure is not deviated. Further, the technical scope of the present disclosure is not limited to these embodiments, but relates to the inventions described in the claims and their equivalents.

[0109] For example, in the above-described embodiments, the process of finding the eye opening degree of the driver is one example, and is not limited thereto. Further, in the above-described embodiments, the process of determining the movement of the face of the driver is one example, and is not limited thereto.

Claims

1. A determination device, comprising: a first determination section that determines whether or not an open eye degree of a driver is equal to or less than a reference open eye degree; a second determination section that determines, based on information that represents movement of a face of the driver, whether or not the face of the driver indicates movement in a right direction that exceeds a first reference and, continuously with the movement, the face of the driver indicates movement in a left direction that exceeds a second reference; and a third determination section that determines that the driver is not dozing off when the first determination section determines that the open eye degree of the driver is equal to or less than the reference open eye degree, the second determination section determines that the face of the driver indicates movement in the right direction that exceeds the first reference and, continuously with the movement, the face of the driver indicates movement in the left direction that exceeds the second reference, and determines that the driver is dozing off when the first determination section determines that the open eye degree of the driver is equal to or less than the reference open eye degree and the second determination section determines that the face of the driver does not indicate movement in the right direction that exceeds the first reference and, continuously with the movement, the face of the driver does not indicate movement in the left direction that exceeds the second reference.

2. The determination device according to claim 1, wherein the information that represents movement of the face of the driver is represented by an angle of a horizontal direction of an orientation of the face of the driver, an angular velocity of the horizontal direction of the orientation of the face of the driver, or an angular acceleration of the horizontal direction of the orientation of the face of the driver.

3. The determination device according to claim 1, wherein the third determination section determines that the driver is not dozing off when the second determination section determines that the face of the driver indicates movement in the right direction that exceeds the first reference and, continuously with the movement, the face of the driver indicates movement in the left direction that exceeds the second reference within a reference time from a point in time when it is determined that the driver is not dozing off.

4. The determination device according to claim 1, further comprising: a fourth determination section that determines, based on movement of the face of the driver, whether or not an orientation of the face of the driver indicates movement in an upward direction that exceeds a third reference or indicates movement in a downward direction that exceeds a fourth reference, the third determination section determines that the driver is dozing off when the first determination section determines that the open eye degree of the driver is equal to or less than the reference open eye degree and the fourth determination section determines that the face of the driver indicates movement in the upward direction that exceeds the third reference or indicates movement in the downward direction that exceeds the fourth reference, even when the second determination section determines that the face of the driver indicates movement in the right direction that exceeds the first reference and, continuously with the movement, the face of the driver indicates movement in the left direction that exceeds the second reference.

5. A determination program for causing a processor to execute: determining whether or not an open eye degree of a driver is equal to or less than a reference open eye degree; determining, based on information representing movement of a face of the driver, whether or not the face of the driver indicates movement in a right direction exceeding a first reference and, continuously with the movement, the face of the driver indicates movement in a left direction exceeding a second reference; and in a case where it is determined that the open eye degree of the driver is equal to or less than the reference open eye degree and it is determined that the face of the driver indicates movement in the right direction exceeding the first reference and, continuously with the movement, the face of the driver indicates movement in the left direction exceeding the second reference, determining that the driver is not dozing off, and in a case where it is determined that the open eye degree of the driver is equal to or less than the reference open eye degree and it is determined that the face of the driver does not indicate movement in the right direction exceeding the first reference and, continuously with the movement, the face of the driver does not indicate movement in the left direction exceeding the second reference, determining that the driver is dozing off.

6. A determination method comprising, by a determination device: determining whether or not an open eye degree of a driver is equal to or less than a reference open eye degree; determining, based on information representing movement of a face of the driver, whether or not the face of the driver indicates movement in a right direction exceeding a first reference and, continuously with the movement, the face of the driver indicates movement in a left direction exceeding a second reference; and in a case where it is determined that the open eye degree of the driver is equal to or less than the reference open eye degree and it is determined that the face of the driver indicates movement in the right direction exceeding the first reference and, continuously with the movement, the face of the driver indicates movement in the left direction exceeding the second reference, determining that the driver is not dozing off, and in a case where it is determined that the open eye degree of the driver is equal to or less than the reference open eye degree and it is determined that the face of the driver does not indicate movement in the right direction exceeding the first reference and, continuously with the movement, the face of the driver does not indicate movement in the left direction exceeding the second reference, determining that the driver is dozing off.

Images