Determination device, determination computer program, and determination method

Abstract

To provide a determination device capable of determining whether a driver is dozing without erroneously detecting that the driver is smiling when the driver is dozing.SOLUTION: A determination device includes a determination unit that determines that a driver is not dozing when it is determined that the degree of eye opening of the driver is equal to or less than a reference degree of eye opening, and that the direction of the driver's face shows a movement to the right that exceeds a first reference and that, consecutive to that movement, the direction of the driver's face shows a movement to the left that exceeds a second reference, and that the driver is dozing when it is determined that the degree of eye opening of the driver is equal to or less than the reference degree of eye opening, and that the direction of the driver's face shows a movement to the right that exceeds the first reference and that, consecutive to that movement, the direction of the driver's face does not show a movement to the left that exceeds the second reference.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a determination device, a determination computer program, and a determination method. [Background technology]

[0002] Conventionally, a monitoring device mounted on a vehicle monitors the state of the driver, for example, whether the driver is falling asleep at the wheel.

[0003] The monitoring device checks the degree of eye opening based on the image of the driver's face to determine whether the driver is dozing. If the monitoring device detects that the driver's eyes are closed for a predetermined period of time, it determines that the driver is dozing and issues a warning to the driver.

[0004] For example, Patent Document 1 proposes a technology for determining whether a driver is drowsy based on eyelid movement and the change in the angle of the face immediately after the eyelid movement occurs. In Patent Document 1, if the angle of the face moves to the left or right by more than a predetermined amount immediately after the eyelids close, it is determined that the driver is not drowsy. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-048531 Summary of the Invention [Problem to be solved by the invention]

[0006] However, since drivers may fall asleep with their faces facing in one direction, the technology proposed in Patent Document 1 may not be able to detect that the driver is drowsy.

[0007] In addition, even when the driver is smiling, the degree of eye opening may be below the threshold. Therefore, simply detecting that the face angle has moved to the left or right by more than a predetermined amount immediately after the eyelids are closed may not accurately detect that the driver is smiling.

[0008] Therefore, an object of the present disclosure is to provide a determination device that can determine whether a driver is dozing off without erroneously detecting that the driver is smiling. [Means for solving the problem]

[0009] (1) According to one embodiment, a determination device is provided. This determination device is characterized by having a first determination unit that determines whether the degree of eye opening of the driver is equal to or less than a reference degree of eye opening; a second determination unit that determines, based on information representing the movement of the driver's face, whether the direction of the driver's face moves to the right exceeding a first reference and, subsequently, moves to the left exceeding a second reference; and a third determination unit that determines that the driver is not dozing when the first determination unit determines that the degree of eye opening of the driver is equal to or less than the reference degree of eye opening and the second determination unit determines that the direction of the driver's face moves to the right exceeding the first reference and, subsequently, moves to the left exceeding the second reference, and determines that the driver is dozing when the first determination unit determines that the degree of eye opening of the driver is equal to or less than the reference degree of eye opening and the second determination unit determines that the direction of the driver's face does not move to the right exceeding the first reference and, subsequently, moves to the left exceeding the second reference.

[0010] (2) In the determination device of (1), it is preferable that the information representing the movement of the driver's face is represented by the horizontal angle of the driver's face direction, the horizontal angular velocity of the driver's face direction, or the horizontal angular acceleration of the driver's face direction.

[0011] (3) In the determination device of (1) or (2), it is preferable that the third determination unit determines that the driver is not dozing when the second determination unit determines that the driver's face has moved to the right exceeding the first standard and, immediately following that movement, the driver's face has moved to the left exceeding the second standard within a reference time from the time when it was determined that the driver is not dozing.

[0012] (4) In any of the determination devices of (1) to (3), the determination device further includes a fourth determination unit that determines whether the driver's facial orientation has moved upward beyond the third standard or whether the driver's facial orientation has moved downward beyond the fourth standard based on the movement of the driver's face. When the first determination unit determines that the degree of eye opening of the driver is equal to or less than the standard degree of eye opening and the fourth determination unit determines that the driver's facial orientation has moved upward beyond the third standard or has moved downward beyond the fourth standard, it is preferable that the third determination unit determines that the driver is dozing off even when the second determination unit determines that the driver's facial orientation has moved rightward beyond the first standard and, following that movement, has moved leftward beyond the second standard.

[0013] (5) According to another embodiment, there is provided a computer program for determination, which causes a processor to execute the following steps: determine whether a driver's eye opening degree is equal to or less than a reference eye opening degree; determine whether the driver's face has moved to the right exceeding a first reference and subsequently moved to the left exceeding a second reference based on information representing the driver's facial movement; determine that the driver is not dozing when it is determined that the driver's eye opening degree is equal to or less than the reference eye opening degree and that the driver's face has moved to the right exceeding the first reference and subsequently moved to the left exceeding the second reference; and determine that the driver is dozing when it is determined that the driver's eye opening degree is equal to or less than the reference eye opening degree and that the driver's face has not moved to the right exceeding the first reference and subsequently moved to the left exceeding the second reference.

[0014] (6) According to yet another embodiment, there is provided a determination method, which includes the steps of: a monitoring device determining whether a driver's eye opening degree is equal to or less than a reference eye opening degree; determining, based on information representing the driver's facial movement, whether the driver's facial orientation has moved to the left exceeding a first reference and, subsequently, moved to the right exceeding a second reference; determining that the driver is not dozing when it is determined that the driver's eye opening degree is equal to or less than the reference eye opening degree and that the driver's facial orientation has moved to the left exceeding the first reference and, subsequently, moved to the right exceeding the second reference; and determining that the driver is dozing when it is determined that the driver's eye opening degree is equal to or less than the reference eye opening degree and that the driver's facial orientation has not moved to the left exceeding the first reference and, subsequently, moved to the right exceeding the second reference. [Effects of the Invention]

[0015] The determination device according to the present disclosure can determine whether the driver is dozing off without erroneously detecting that the driver is smiling. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 2 is a diagram illustrating an outline of the operation of the monitoring device according to the first embodiment. [Figure 2] 1 is a schematic configuration diagram of a vehicle in which a monitoring device of a first embodiment is mounted. [Figure 3] 4 is an example of an operational flowchart relating to a monitoring process of the monitoring device of the first embodiment. [Figure 4] FIG. 10 is a diagram showing changes in face direction representing the movement of the driver's face. [Figure 5] FIG. 10 is a diagram showing changes in angular velocity representing the movement of the driver's face. [Figure 6] 10 is an example of an operational flowchart relating to a monitoring process of the monitoring device of the second embodiment. [Figure 7] 10 is another example of an operational flowchart relating to the monitoring process of the monitoring device of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Fig. 1 is a diagram illustrating an outline of the operation of the monitoring device of the first embodiment. As shown in Fig. 1, a vehicle 10 has a monitoring device 11. In a passenger compartment 10A of the vehicle 10, a driver 30 sits in a driver's seat 40. A monitoring camera 2 is disposed on the dashboard.

[0018] The monitoring device 11 monitors the driver 30 based on the monitoring images acquired by the monitoring camera 2. The monitoring device 11 is an example of a determination device. The vehicle 10 may be an autonomous vehicle.

[0019] The monitoring camera 2 can acquire an image showing the face of the driver 30. The monitoring device 11 detects the degree to which the eyes of the driver 30 are open, based on the monitoring image acquired by the monitoring camera 2. The monitoring device 11 also detects information showing the movement of the face of the driver 30, based on the monitoring image.

[0020] The monitoring device 11 determines whether the driver 30 is dozing off based on information indicating the degree of eye opening of the driver 30 and the facial movement of the driver 30. If it is determined that the driver 30 is dozing off, the monitoring device 11 notifies a warning to the driver 30 via a user interface (UI) 4.

[0021] The monitoring device 11 determines whether the driver 30 has his / her eyes closed based on the degree to which the driver 30 has their eyes open. Here, if the driver 30 is smiling, a state in which the degree of eye opening is low may be detected for a predetermined period of time. To prevent the driver from being mistakenly determined to be dozing, it is desirable to be able to correctly determine whether the driver is smiling or dozing.

[0022] As a result of research, the inventors of the present disclosure have found that when people smile, they continuously change the direction of their faces from left to right. When people sleep, they may turn their faces to one side to make their heads more comfortable, but they do not continuously change the direction of their faces from left to right when they are sleeping. Continuously changing the direction of their faces from left to right is a characteristic of when a person is smiling.

[0023] Therefore, when it is determined that the driver 30 has his eyes closed, the monitoring device 11 determines whether the driver 30 is continuously changing the direction of his face from side to side based on information indicating the movement of his face.

[0024] If it is determined that the degree of eye opening of the driver 30 is equal to or less than the reference eye opening degree, the monitoring device 11 checks the facial movement of the driver 30. If it is determined that the direction of the driver 30's face moves to the right exceeding a first reference and, subsequently, the direction of the driver 30's face moves to the left exceeding a second reference, the monitoring device 11 determines that the driver 30 is not dozing.

[0025] Even if it is determined that the driver 30 has his / her eyes closed, if the direction of his / her face is continuously changing from side to side, the driver 30 is smiling. By determining whether the direction of his / her face is continuously changing from side to side, it is possible to prevent the smiling face of the driver 30 from being mistakenly determined as being dozing.

[0026] Furthermore, if it is determined that the degree of eye opening of the driver 30 is equal to or less than the reference degree of eye opening, and if it is determined that the direction of the face of the driver 30 moves to the right exceeding a first reference and does not subsequently move to the left exceeding a second reference, the monitoring device 11 determines that the driver 30 is dozing.

[0027] That is, if it is determined that the driver 30 has his eyes closed and the direction of his face is not continuously changing from side to side, it is estimated that the driver 30 is dozing off.

[0028] When the monitoring device 11 determines that the driver 30 is dozing, it notifies the driver 30 of a warning via the UI 4 and increases the degree to which the driver is involved in driving.

[0029] As described above, the monitoring device 11 of this embodiment can determine whether the driver 30 is dozing off without erroneously detecting that the driver 30 is smiling.

[0030] 2 is a schematic diagram of a vehicle 10 in which a monitoring device 11 of this embodiment is installed. The vehicle 10 has a monitoring camera 2, a user interface (UI) 4, a monitoring device 11, and the like.

[0031] The monitoring camera 2, the UI 4, and the monitoring device 11 are communicably connected via an in-vehicle network 12 that complies with a standard such as a controller area network.

[0032] The surveillance camera 2 is disposed in the vehicle interior 10A so as to be able to acquire surveillance images including the face of the driver 30 who drives the vehicle 10. The surveillance camera 2 is an example of an image acquisition unit. For example, the surveillance camera 2 is disposed so as to be able to acquire images of the vicinity of the driver's seat 40. As shown in FIG. 1, the surveillance camera 2 is disposed, for example, on the dashboard.

[0033] The surveillance camera 2 acquires surveillance images at image acquisition times having a predetermined cycle, for example. Every time the surveillance camera 2 acquires a surveillance image, it outputs the surveillance image and the image acquisition time to the monitoring device 11 via the in-vehicle network 12. The predetermined cycle can be, for example, 0.1 to 0.5 seconds.

[0034] The surveillance camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements, such as a CCD or C-MOS, that are sensitive to infrared light, and an imaging optical system that forms an image of the area to be imaged on the two-dimensional detector. The surveillance camera 2 preferably has a light projector in addition to the two-dimensional detector. The light projector is, for example, two near-infrared LEDs arranged on both sides of the imaging optical system. By irradiating the driver 30 with near-infrared light, the face of the driver 30 can be captured without causing discomfort to the driver 30, even in low-light conditions, such as at night.

[0035] The UI4 is an example of a notification unit. The UI4 is controlled by the monitoring device 11 to notify the driver 30 of warnings and the like. The UI4 has a display device 4a such as a liquid crystal display or a touch panel for displaying warnings and the like. The UI4 may also have an audio output device (not shown) for notifying the driver 30 of warnings and the like. The UI4 also has, for example, a touch panel or operation buttons as an input device for inputting operation information from the driver 30 to the vehicle 10. The UI4 outputs the input operation information to the monitoring device 11 via the in-vehicle network 12.

[0036] The monitoring device 11 executes detection processing, determination processing, and control processing. To this end, the monitoring device 11 includes a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 includes an interface circuit for connecting the monitoring device 11 to the in-vehicle network 12.

[0037] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores various data and computer programs of applications used in information processing executed by the processor 23. The memory 22 stores surveillance images input from the surveillance camera 2 in association with the image acquisition time.

[0038] All or part of the functions of the monitoring device 11 are functional modules realized by, for example, a computer program running on the processor 23. The processor 23 has a detection unit 231, a determination unit 232, and a control unit 233. The determination unit 232 is an example of a first determination unit, a second determination unit, and a third determination unit. The functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23.

[0039] The processor 23 has one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic unit, a numerical calculation unit, or a graphics processing unit. The monitoring device 11 is, for example, an electronic control unit (ECU).

[0040] The detection unit 231 determines the degree of eye opening of the driver 30 based on a plurality of monitoring images at eye opening degree detection times having a predetermined cycle. The cycle can be set to, for example, 0.1 to 5 seconds.

[0041] The detection unit 231 has a classifier that has been trained to receive a surveillance image and detect eye regions. The classifier receives a surveillance image and detects eye regions included in the surveillance image.

[0042] The classifier is, for example, a deep neural network (DNN) having multiple layers connected in series from the input side to the output side. Images including eyes are input to the DNN in advance as training data, and the DNN operates as a classifier for detecting eye regions by learning. Alternatively, a machine learning model such as a support vector machine or a random forest may be used as the classifier.

[0043] Furthermore, the detection unit 231 determines the distance from the surveillance camera 2 to the face of the driver 30. For example, the detection unit 231 refers to the relationship between the size and distance of a typical human face depicted in an image, and determines the distance between the surveillance camera 2 and the face of the driver 30 based on the size of the face of the driver 30 depicted in the surveillance image. The area of ​​the face of the driver 30 in the surveillance image is detected using a classifier trained to detect face areas.

[0044] The detection unit 231 determines the degree of eye opening based on the distance from the monitoring camera 2 to the face of the driver 30 and the number of pixels between the upper eyelid and the lower eyelid in the monitoring image. For example, the detection unit 231 may determine the maximum number of pixels between the upper eyelid and the lower eyelid. The degree of eye opening is expressed in units of mm, for example. The detection unit 231 notifies the determination unit 232 of the degree of eye opening of each of the driver's right eye and left eye.

[0045] The detection unit 231 detects information representing the facial movement of the driver 30 based on the monitored image at information detection times having a predetermined cycle. The cycle can be, for example, 0.1 to 1 second. The detection unit 231 notifies the determination unit 232 of the information representing the facial movement of the driver 30.

[0046] Information representing the movement of the face of the driver 30 is represented, for example, by the horizontal angle of the direction of the face of the driver 30, the horizontal angular velocity of the direction of the face of the driver 30, or the horizontal angular acceleration of the direction of the face of the driver 30.

[0047] The direction of the face of the driver 30 is represented, for example, by the horizontal angle between the traveling direction of the vehicle 10 and the direction in which the face of the driver 30 is facing. For example, assuming that the traveling direction of the vehicle 10 is 0 degrees, the direction of the face of the driver 30 facing left is represented by an angle of 0 degrees to -180 degrees, and the direction of the face of the driver 30 facing right is represented by an angle of 0 degrees to 180 degrees.

[0048] The detection unit 231 has a classifier that has been trained to detect specific facial features, such as the corners of the eyes, the inner corners of the eyes, and the corners of the mouth, from an image. The detection unit 231 inputs a surveillance image into this classifier, thereby identifying the positions of the specific features included in the surveillance image. The detection unit 231 then compares the positions of the specific features detected from the surveillance image with a standard 3D model of a face. The detection unit 231 then detects, as the angle of the face orientation in the surveillance image, the angle of the face orientation in the 3D model when the position of each feature most closely matches the position of the feature detected in the surveillance image.

[0049] The classifier can be, for example, a deep neural network (DNN) with multiple layers connected in series from the input side to the output side. By inputting facial images containing predetermined facial features as training data into the DNN in advance and performing learning, the DNN operates as a classifier that identifies the positions of the predetermined facial features.

[0050] The detection unit 231 obtains a time-series change in the horizontal angle of the face direction of the driver 30. The detection unit 231 may obtain the face direction of the driver 30 based on the line of sight of the driver 30.

[0051] Furthermore, the detection unit 231 may obtain the horizontal angular velocity of the face direction of the driver 30 as the amount of change per unit time in the horizontal angle of the face direction of the driver 30. In this way, the detection unit 231 obtains the time-series change in the horizontal angular velocity of the face direction of the driver 30.

[0052] Furthermore, the detection unit 231 may obtain the horizontal angular acceleration of the face direction of the driver 30 as the amount of change per unit time in the horizontal angular velocity of the face direction of the driver 30. In this way, the detection unit 231 obtains the time-series change in the horizontal angular acceleration of the face direction of the driver 30.

[0053] Furthermore, the detection unit 231 may detect vertical movement of the face of the driver 30. The vertical movement of the face direction of the driver 30 is represented, for example, by the vertical angle between the traveling direction of the vehicle 10 and the direction in which the driver 30 is facing. Furthermore, the detection unit 231 may obtain the angular velocity or angular acceleration of the face direction of the driver 30 in the vertical direction.

[0054] For example, with the direction of travel of the vehicle 10 being 0 degrees, the direction of the face of the driver 30 facing upward is expressed as an angle from 0 degrees to 90 degrees, and the direction of the face of the driver 30 facing downward is expressed as an angle from 0 degrees to -90 degrees.

[0055] Fig. 3 is an example of an operational flowchart relating to the monitoring process of the monitoring device 11 of this embodiment. Next, the monitoring process of the monitoring device 11 will be described below with reference to Fig. 3. The monitoring device 11 executes the monitoring process in accordance with the operational flowchart shown in Fig. 3 at a monitoring time having a predetermined cycle.

[0056] First, the determination unit 232 calculates the average value of the eye opening degree of the driver 30 (step S101). For example, the determination unit 232 calculates the average value of the eye opening degree obtained within a recent fixed period for each of the left and right eyes of the driver 30. The recent fixed period can be 10 to 30 seconds.

[0057] Next, the determination unit 232 determines whether the driver 30 has his / her eyes closed based on the degree of eye opening of the driver 30 (step S102). For example, if the average values ​​of the degrees of eye opening of the driver 30's left and right eyes are equal to or less than a reference degree of eye opening, the determination unit 232 determines that the driver 30 has his / her eyes closed. The reference degree of eye opening can be set to, for example, 3 mm. Typically, the degree of eye opening of a person is in the range of 10 mm to 30 mm.

[0058] On the other hand, if at least one of the average values ​​of the eye opening degrees of the left and right eyes of the driver 30 is not equal to or less than the reference eye opening degree, the determining unit 232 determines that the driver 30 is not in an eye-closed state.

[0059] If it is determined that the driver 30 has his / her eyes closed (step S102-Yes), the determination unit 232 determines whether the direction of the driver 30's face has moved to the right exceeding a first criterion based on information representing the movement of the driver 30's face during a predetermined determination period (step S103). For example, the determination period may be set to 1 to 5 seconds. The determination period may start from the time when it is determined that the driver 30 has his / her eyes closed. Alternatively, the determination period may include the time when it is determined that the driver 30 has his / her eyes closed.

[0060] When the information representing the facial movement of the driver 30 is expressed as an angle, the first reference may be, for example, -10 degrees. When the information representing the facial movement of the driver 30 is expressed as an angular velocity, the first reference may be, for example, -2.5 degrees / second. When the information representing the facial movement of the driver 30 is expressed as an angular acceleration, the first reference may be, for example, -2.5 degrees / second. 2 It can be said that:

[0061] The determination unit 232 checks, from the start time of the determination period, information representing the facial movement of the driver 30. If the information representing the facial movement of the driver 30 has a value exceeding a first criterion, the determination unit 232 determines that the orientation of the driver 30's face has moved to the right exceeding the first criterion.

[0062] If the direction of the driver's face indicates a movement to the right that exceeds a first standard (hereinafter also referred to as a first movement) (step S103-Yes), the determination unit 232 determines whether or not the direction of the driver's face indicates a movement to the left that exceeds a second standard, consecutive to the first movement, based on information representing the movement of the driver's face in a determination period (step S104). The determination period is the same period as the determination of the first movement. The information representing the movement of the driver's face is the same information as that for the first movement.

[0063] The determination unit 232 checks the information representing the facial movement of the driver 30 from the start time of the determination period. If the information representing the facial movement of the driver 30 has a value exceeding a second standard following the first movement, the determination unit 232 determines that the direction of the driver 30's face has moved to the left exceeding the second standard (hereinafter also referred to as the second movement). The second movement may appear before the first movement or after the first movement.

[0064] When the information representing the facial movement of the driver 30 is expressed as an angle, the second reference may be, for example, +10 degrees. When the information representing the facial movement of the driver 30 is expressed as an angular velocity, the second reference may be, for example, +2.5 degrees / second. When the information representing the facial movement of the driver 30 is expressed as an angular acceleration, the second reference may be, for example, +2.5 degrees / second. 2 The absolute value of the second criterion may be the same as or different from the first criterion.

[0065] If the direction of the driver's face 30 moves to the left beyond the second standard (step S104-Yes) or if it is determined that the driver 30 is not in a closed-eye state (step S102-No), the judgment unit 232 determines that the driver 30 is not dozing (step S105) and ends the series of processes.

[0066] On the other hand, if the direction of the driver's face does not move to the right beyond the first standard (step S103-No), or if the direction of the driver's face does not move to the left beyond the second standard (step S104-No), the judgment unit 232 judges that the driver 30 is dozing off (step S106).

[0067] Note that a process of determining whether or not the eyes are closed may be performed between step S103 and step S104 and step S106. The determination unit 232 may proceed to step S106 after determining that the driver 30 has the eyes closed for a predetermined time (for example, 3 to 10 seconds). If it is determined that the driver 30 does not have the eyes closed during this time, the series of processes ends.

[0068] When the direction of the driver's face 30 moves to the right and exceeds the first standard, and does not subsequently move to the left and exceeds the second standard, this includes the absence of the first and second movements, the presence of the first movement but not the second movement, and the absence of the first movement but the presence of the second movement.

[0069] Next, the control unit 233 notifies the driver 30 of a warning via the UI 4 (step S107), and ends the series of processes. For example, the control unit 233 displays the warning on the display device 4a and outputs a loud warning sound via the UI 4. This causes the monitoring device 11 to increase the degree to which the driver 30 is involved in driving.

[0070] 4 is a diagram showing changes in the facial direction of the driver 30, representing the facial movement of the driver 30. The vertical axis represents the horizontal angle of the facial direction, and the horizontal axis represents time. At time zero, the driver 30 is determined to have his eyes closed. The determination unit 232 determines the facial movement of the driver 30 over a period Tm from time zero.

[0071] The angle representing the face direction exceeds the first reference +Th1 at time T1, and then exceeds the second reference -Th1 at time T2. The second action, in which the angle exceeds the second reference -Th1 at time T2, is continuous with the first action, in which the angle exceeds the first reference +Th1 at time T1. Time T2 is within the period Tm. It is estimated that the driver 30 is smiling. Therefore, the determination unit 232 determines that the driver 30 is not dozing.

[0072] 5 is a diagram showing changes in angular velocity representing the facial movement of driver 30. The vertical axis represents the horizontal angular velocity of the facial orientation, and the horizontal axis represents time. At time zero, it is determined that driver 30 has his eyes closed. The determination unit 232 determines the facial movement of driver 30 over a period Tm from time zero.

[0073] The angular velocity representing the facial movement exceeds the first standard +Th2 at time T1, and then exceeds the second standard -Th2 at time T2. The second movement, in which the angular velocity exceeds the second standard -Th2 at time T2, is continuous with the first movement, in which the angular velocity exceeds the first standard +Th2 at time T1. Time T2 is within the period Tm. It is estimated that the driver 30 is smiling. Therefore, the determination unit 232 determines that the driver 30 is not dozing.

[0074] To express the angle representing the face direction described above, it is necessary to set the traveling direction of the vehicle 10 as a reference for the angle. On the other hand, since the angular velocity is expressed as the amount of change in angle per unit time, there is no need to set a reference, and therefore using the angular velocity makes processing easier. Processing is also easier when the movement of the driver 30's face is expressed by angular acceleration.

[0075] According to the monitoring device of this embodiment described above in detail, it is possible to determine whether the driver is dozing off without erroneously detecting that the driver is smiling.

[0076] Next, monitoring devices according to second and third embodiments disclosed in this specification will be described below with reference to Figures 6 and 7. For the monitoring device according to this embodiment, the description of the monitoring device according to the first embodiment described above applies as appropriate to the points not specifically described.

[0077] 6 is an example of an operational flowchart relating to the monitoring process of the monitoring device 11 of the second embodiment. This embodiment differs from the first embodiment in that the processes of steps S206 to S208 are added. The processes of steps S201 to S205, S209, and S210 are the same as the processes of steps S101 to S107 described above.

[0078] In this embodiment, after it is determined that the driver 30 is not dozing (step S205), the determination unit 232 determines whether the direction of the driver 30's face has moved to the right exceeding the first criterion (step S206).

[0079] The determination unit 232 determines whether the driver 30's face has moved to the right exceeding a first standard based on information representing the driver 30's facial movement during the determination period. For example, this determination period can be set to 1 to 5 seconds. The determination period may start from the time when it is determined that the driver 30 is not dozing.

[0080] If the direction of the driver's face 30 shows a movement to the right that exceeds the first standard (step S206-Yes), the judgment unit 232 judges whether the direction of the driver's face 30 shows a movement to the left that exceeds the second standard, consecutive to the first action, based on information representing the movement of the driver's face 30 during the judgment period (step S207).

[0081] If the direction of the driver 30's face moves to the left beyond the second standard (step S207-Yes), the determination unit 232 determines whether or not the reference time has passed since it was determined that the driver 30 was not dozing (step S208). The determination unit 232 determines whether or not the time when the second movement was confirmed is within the reference time from the time when it was determined that the driver 30 was not dozing. The reference time can be set to, for example, 3 to 10 seconds.

[0082] If it is within the reference time (step S208-Yes), the determination unit 232 determines that the driver 30 is not dozing (step S205). After that, the process proceeds to step S206.

[0083] On the other hand, if the direction of the face of the driver 30 does not move to the right beyond the first reference (step S206-No), if the direction of the face of the driver 30 does not move to the left beyond the second reference (step S207-No), or if the time has not passed within the reference time (step S208-No), the series of processes ends. Since it is estimated that the driver 30 is not smiling, the monitoring process starts again from monitoring the closed eye state.

[0084] The monitoring process of this embodiment will be described below with reference to Figures 4 and 5. In the example shown in Figure 4, it is first determined that the driver 30 is not dozing at time T2. Therefore, the determination unit 232 continues to determine the movement of the face of the driver 30 without determining whether the eyes are closed.

[0085] The angle representing the face direction exceeds the first reference +Th1 at time T3, and then exceeds the second reference -Th1 at time T4. The second movement, whose angle exceeds the second reference -Th1 at time T4, is continuous with the first movement, whose angle exceeds the first reference +Th1 at time T3.

[0086] Time T4 is within the reference time Tn from time T2 at which it was determined that the driver 30 was not dozing, so the determination unit 232 determines that the driver 30 is not dozing.

[0087] 5, it is first determined that the driver 30 is not dozing at time T2. Therefore, the determination unit 232 continues to determine the movement of the face of the driver 30 without determining whether the eyes are closed.

[0088] The angular velocity representing the facial movement exceeds the first standard +Th2 at time T3, and then exceeds the second standard −Th2 at time T4. The second movement, whose angular velocity exceeds the second standard −Th2 at time T4, is continuous with the first movement, whose angular velocity exceeds the first standard +Th2 at time T3.

[0089] Time T4 is within the reference time Tn from time T2 at which it was determined that the driver 30 was not dozing, so the determination unit 232 determines that the driver 30 is not dozing.

[0090] In this embodiment, if it is determined that the driver 30 is not dozing, it is estimated that the driver 30 is smiling. Therefore, in this embodiment, the determination of the facial movement of the driver 30 continues without determining whether the eyes are closed.

[0091] As described above in detail, the monitoring device of this embodiment can reduce the processing load of the monitoring device. Furthermore, the monitoring device of this embodiment has the same effects as the first embodiment described above.

[0092] 7 is an example of an operational flowchart relating to the monitoring process of the monitoring device 11 of the third embodiment. This embodiment differs from the first embodiment in that the process of step S305 is added. The processes of steps S301 to S304 and S306 to S308 are the same as those of steps S101 to S107 described above.

[0093] 306 If the direction of the driver's face 30 moves to the left and exceeds the second standard (step S304-Yes), the judgment unit 232 judges, based on the movement of the driver's face 30, whether the direction of the driver's face 30 moves upward and exceeds the third standard, or whether the direction of the driver's face 30 moves downward and exceeds the fourth standard (step S306).

[0094] The determination unit 232 acquires information representing the vertical movement of the face of the driver 30 from the detection unit 231. The information representing the vertical movement of the face of the driver 30 is represented by an angle, an angular velocity, or an angular acceleration.

[0095] Based on the information representing the vertical movement of the face of the driver 30, the determination unit 232 determines whether the orientation of the face of the driver 30 has moved upward beyond a third criterion or whether the orientation of the face of the driver 30 has moved downward beyond a fourth criterion. The above description of the process for determining the information representing the horizontal movement of the face of the driver 30 can be applied as appropriate to determining the information representing the vertical movement of the face of the driver 30. For example, the third criterion can be 10 degrees, and the fourth criterion can be -10 degrees.

[0096] If the direction of the driver's face 30 shows an upward movement exceeding the third standard, or if the direction of the driver's face 30 shows a downward movement exceeding the fourth standard (step S306-Yes), the judgment unit 232 judges that the driver 30 is dozing off (step S307).

[0097] If the direction of the driver's face 30 shows upward or downward movement exceeding the standard, it is highly likely that the driver 30 is dozing, and therefore it is determined that the driver 30 is dozing, regardless of the horizontal movement of the driver's face.

[0098] As described above in detail, the monitoring device of this embodiment can reliably determine whether the driver 30 is dozing. Furthermore, the monitoring device of this embodiment has the same effects as the first embodiment described above.

[0099] In the present disclosure, the determination device, the determination computer program, and the determination method of the above-described embodiments can be modified as appropriate without departing from the spirit of the present disclosure. Furthermore, the technical scope of the present disclosure is not limited to those embodiments, but extends to the inventions set forth in the claims and their equivalents.

[0100] For example, in the above-described embodiment, the process of determining the degree of eye opening of the driver is an example and is not limited to this. Also, in the above-described embodiment, the process of determining the movement of the driver's face is an example and is not limited to this. [Explanation of symbols]

[0101] 2. Surveillance cameras 4 User Interface 4a Display device 10 vehicles 11 Monitoring equipment 21 Communication Interface 22 Memory 23 processors 231 Detector 232 Judgment section 233 Control Unit 12 In-vehicle network

Claims

1. a first determination unit that determines whether the degree of eye opening of the driver is equal to or less than a reference degree of eye opening; a second determination unit that determines whether or not the direction of the driver's face has moved to the right and exceeded a first standard, and whether or not the direction of the driver's face has moved to the left and exceeded a second standard, consecutively to the first standard, based on information indicating the movement of the driver's face; a third determination unit that determines that the driver is not dozing when the first determination unit determines that the degree of eye opening of the driver is equal to or less than the reference degree of eye opening and when the second determination unit determines that the direction of the driver's face moves to the right exceeding the first standard and, consecutively to the movement, the direction of the driver's face moves to the left exceeding the second standard; and that the driver is dozing when the first determination unit determines that the degree of eye opening of the driver is equal to or less than the reference degree of eye opening and when the second determination unit determines that the direction of the driver's face does not move to the right exceeding the first standard and, consecutively to the movement, the direction of the driver's face does not move to the left exceeding the second standard.

2. 2. The determination device according to claim 1, wherein the information representing the movement of the driver's face is represented by a horizontal angle of the driver's face direction, a horizontal angular velocity of the driver's face direction, or a horizontal angular acceleration of the driver's face direction.

3. 2. The determination device according to claim 1, wherein the third determination unit determines that the driver is not dozing when the second determination unit determines that the driver's face direction has moved to the right exceeding the first standard and, immediately following that movement, the driver's face direction has moved to the left exceeding the second standard within a reference time from the time when it was determined that the driver is not dozing.

4. a fourth determination unit that determines whether or not the orientation of the driver's face has moved upward and exceeded a third standard, or whether or not the orientation of the driver's face has moved downward and exceeded a fourth standard, based on the movement of the driver's face; 2. The determination device according to claim 1, wherein the third determination unit determines that the driver is dozing when the first determination unit determines that the degree of eye opening of the driver is equal to or less than the reference degree of eye opening and the fourth determination unit determines that the direction of the driver's face has moved upward exceeding the third reference or moved downward exceeding the fourth reference, even if the second determination unit determines that the direction of the driver's face has moved rightward exceeding the first reference and, immediately following that movement, has moved leftward exceeding the second reference.

5. Determine whether the degree of eye opening of the driver is equal to or less than a reference degree of eye opening; Based on the information representing the movement of the driver's face, it is determined whether or not the direction of the driver's face has moved to the right exceeding a first standard and, following the movement, the direction of the driver's face has moved to the left exceeding a second standard; When it is determined that the degree of eye opening of the driver is equal to or less than the reference eye opening degree, and when it is determined that the direction of the driver's face moves to the right exceeding the first reference and, subsequently to that movement, the direction of the driver's face moves to the left exceeding the second reference, it is determined that the driver is not dozing; and when it is determined that the degree of eye opening of the driver is equal to or less than the reference eye opening degree, and when it is determined that the direction of the driver's face moves to the right exceeding the first reference and, subsequently to that movement, the direction of the driver's face does not move to the left exceeding the second reference, it is determined that the driver is dozing. A computer program for determination, which causes a processor to execute the following:

6. The monitoring device Determine whether the degree of eye opening of the driver is equal to or less than a reference degree of eye opening; Based on the information representing the movement of the driver's face, it is determined whether or not the direction of the driver's face has moved to the left exceeding a first standard and, following that movement, the direction of the driver's face has moved to the right exceeding a second standard; When it is determined that the degree of eye opening of the driver is equal to or less than the reference eye opening degree, and when it is determined that the direction of the driver's face moves to the left exceeding the first reference and, subsequently to that movement, the direction of the driver's face moves to the right exceeding the second reference, it is determined that the driver is not dozing; and when it is determined that the degree of eye opening of the driver is equal to or less than the reference eye opening degree, and when it is determined that the direction of the driver's face moves to the left exceeding the first reference and, subsequently to that movement, the direction of the driver's face does not move to the right exceeding the second reference, it is determined that the driver is dozing. A determination method comprising:

Images