Occupant monitoring device

Abstract

To provide an occupant monitoring device accurately detecting presence / absence of blinks of a vehicle occupant.SOLUTION: An occupant monitoring device 1 includes: an acquisition part 11 acquiring an eye-opening distance of an occupant on a vehicle; a detection part 12 detecting blinks of the occupant on the basis of a comparison result between a threshold value for blink determination and an amount of change in an eye-opening distance per a unit time; and a setting part 13 setting the threshold value for blink determination on the basis of the eye-open distance. The setting part 13 sets a corresponding threshold value for each of detection periods, and may decide the threshold value during the detection period of a setting target on the basis of an eye-open speed at the time of blinks during a detection period prior to the last target detection period.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an occupant monitoring device. [Background technology]

[0002] Patent Document 1 discloses an occupant monitoring device that detects whether the driver's eyes are open or closed based on image data of the driver's face, and determines whether the driver is drowsy by using the change in eyelid opening over time as the eyelid opening state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-067137 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to accurately detect whether a driver is dozing off while driving a vehicle, it is desirable to accurately determine whether the driver's eyes are open or closed. For example, it is desirable to accurately determine whether the driver is blinking by detecting the opening and closing of the eyelids as the eye opening and closing state.

[0005] However, eyelid opening and closing movements vary from person to person, making it difficult to accurately set a threshold value suited to an individual in a detection method that compares the degree of eye opening with a threshold and determines that a blink has occurred when the degree of eye opening is equal to or less than the threshold.

[0006] An object of the present invention is to provide an occupant monitoring device that can accurately detect whether or not a vehicle occupant is blinking. [Means for solving the problem]

[0007] The occupant monitoring device of the present invention is characterized by comprising an acquisition unit that acquires the eye-opening distance of an occupant riding in a vehicle, a setting unit that sets a threshold for determining blinking based on the eye-opening distance, and a detection unit that detects blinking of the occupant based on the comparison result between the threshold and the amount of change per unit time of the eye-opening distance. [Effects of the Invention]

[0008] The occupant monitoring device according to the present invention sets a threshold value for determining whether a vehicle occupant is blinking based on the eye-opening distance acquired from the vehicle occupant, thereby enabling a threshold value suitable for the movement of the occupant's eyelids to be set. Since the presence or absence of a blink is determined using this threshold value, the accuracy of blink detection is improved. As a result, the occupant monitoring device can accurately detect whether a vehicle occupant is blinking. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram schematically showing a vehicle equipped with an occupant monitoring device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the occupant monitoring device. [Figure 3] FIG. 3 is a diagram for explaining the amount of change in eye opening distance per unit time. [Figure 4] FIG. 4 is a diagram for explaining the difference between the maximum and minimum values ​​of the eye-opening distance within the detection period. [Figure 5] FIG. 5 is a diagram for explaining the amount of change in the eye-opening distance for each detection period. [Figure 6] FIG. 6 is a diagram for explaining the opening time during blinking. [Figure 7] FIG. 7 is a diagram for explaining a case where labeling is performed. [Figure 8] FIG. 8 is a diagram for explaining setting of a threshold value for determining a blink using blink information in a detection period immediately before the detection target. [Figure 9] FIG. 9 is a diagram for explaining labels that are assigned according to the comparison result between the threshold value and the amount of change in eye-opening distance per unit time. [Figure 10] FIG. 10 is a diagram for explaining the blink duration, closing time, and eye-opening time. [Figure 11] FIG. 11 is a flowchart showing a blink detection flow. [Figure 12] FIG. 12 is a flowchart showing a threshold setting flow. [Figure 13] FIG. 13 is a flowchart showing the labeling flow. [Figure 14] FIG. 14 is a flowchart showing the flow of calculating the opening time. [Figure 15] Figure 15(a) is a graph showing the acquired data on eye opening distance over time, Figure 15(b) is a graph showing the blink detection results of a comparative example, and Figure 15(c) is a graph showing the blink detection results of an embodiment. [Figure 16] FIG. 16 is a flowchart showing a threshold setting flow in the first modified example. [Figure 17] FIG. 17 is a diagram illustrating microsleep. [Figure 18] FIG. 18 is a flowchart showing the flow of the determination process. [Figure 19] FIG. 19 is a table showing the detection results in the second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] An occupant monitoring device according to an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the present invention is not limited to this embodiment. Furthermore, the components in the following embodiments include those that can be easily imagined by a person skilled in the art or that are substantially the same.

[0011] [Embodiment] An occupant monitoring device according to an embodiment will be described with reference to FIGS. 1 to 15. FIG. 1 is a diagram schematically illustrating a vehicle equipped with the occupant monitoring device according to the embodiment. FIG. 2 is a block diagram illustrating the functional configuration of the occupant monitoring device. FIG. 3 is a diagram illustrating the amount of change in eye-opening distance per unit time. FIG. 4 is a diagram illustrating the difference between the maximum and minimum values ​​of the eye-opening distance within a detection period. FIG. 5 is a diagram illustrating the amount of change in eye-opening distance per detection period. FIG. 6 is a diagram illustrating the opening time during a blink. FIG. 7 is a diagram illustrating a case where labeling is performed. FIG. 8 is a diagram illustrating setting a threshold value for determining a blink using blink information from a detection period immediately preceding the detection target. FIG. 9 is a diagram illustrating a label assigned according to a comparison result between a threshold value and the amount of change in eye-opening distance per unit time. FIG. 10 is a diagram illustrating the duration, closing time, and opening time of a blink. FIG. 11 is a flowchart illustrating a blink detection flow. FIG. 12 is a flowchart illustrating a threshold setting flow. FIG. 13 is a flowchart illustrating a labeling flow. Fig. 14 is a flowchart showing the calculation flow of the eye-opening time. Fig. 15(a) is a graph showing the acquired data of the eye-opening distance over time, Fig. 15(b) is a diagram showing the blink detection results in a comparative example, and Fig. 15(c) is a diagram showing the blink detection results in an embodiment.

[0012] As shown in FIG. 1, the occupant monitoring device 1 is provided in a vehicle 3 equipped with a camera 2. The occupant monitoring device 1 is a device that detects the blinking of an occupant riding in the vehicle 3. The camera 2 captures an image of the face of a driver 100 of the vehicle 3. The vehicle 3 is an automobile, and is driven by the driver 100. The camera 2 is disposed in front of the driver 100 in the cabin of the vehicle 3, and captures an image of the eyes 110 of the driver 100. The camera 2 can continuously capture an image of the face of the driver 100 at a predetermined frame rate. For example, the camera 2 can capture video at 30 fbs.

[0013] 2, the occupant monitoring device 1 includes an acquisition unit 11, a detection unit 12, and a setting unit 13. The occupant monitoring device 1 is a computer having an arithmetic circuit, a memory, and an input / output interface. The occupant monitoring device 1 is configured as, for example, an electronic control unit (ECU).

[0014] The acquisition unit 11 acquires the eye opening distance of the driver 100 based on the images acquired from the camera 2. The eye opening distance is an index showing the open / closed state of the eyes 110, and is the distance between the upper eyelid and the lower eyelid. For example, the distance between the points where the upper eyelid and the lower eyelid are furthest apart is the eye opening distance. The acquisition unit 11 calculates the eye opening distance of the driver 100 in each image, for example, by image recognition. In other words, the acquisition unit 11 calculates the eye opening distance for each of the images continuously acquired from the camera 2. When the eyes 110 of the driver 100 are completely closed, the eye opening distance is 0.

[0015] The acquisition unit 11 may calculate the eye opening distance for each of the left eye and the right eye of the driver 100. The eye opening distance may be an average value of the eye opening distance for the left eye and the eye opening distance for the right eye of the driver 100. The acquisition unit 11 may calculate only the eye opening distance for one of the eyes 110 of the driver 100.

[0016] The detection unit 12 detects blinking of the driver 100 based on the eye-opening distance acquired from the driver 100. The detection unit 12 calculates the amount of change in the eye-opening distance per unit time within a predetermined detection period. The detection period is set to, for example, 3 seconds, i.e., the length of 90 frames of image data. The unit time is set to, for example, 33 milliseconds, i.e., the length between frames of image data. The detection unit 12 calculates the amount of change in the eye-opening distance per unit time by subtracting the eye-opening distance of the frame immediately preceding the current frame from the eye-opening distance of the current frame.

[0017] As shown in FIG. 3, the detection unit 12 calculates the difference between the eye-opening distance in frame f2, which is the current frame, and the eye-opening distance in frame f1, which is the frame immediately preceding the current frame. This difference is the distance difference per unit time, i.e., the change amount d of the eye-opening distance per unit time. The detection unit 12 then compares the change amount d of the eye-opening distance per unit time with a threshold B for determining blinks, and determines whether or not a blink is occurring based on the comparison result. In other words, the detection unit 12 does not use a threshold for the degree of eye-opening, but instead detects a single blink by successively determining the upper eyelid lowering stage, the eye closing stage, and the upper eyelid lifting stage based on the change in the eye-opening distance. Note that in this description, the change amount d of the eye-opening distance per unit time may be simply referred to as the change amount d of the eye-opening distance.

[0018] The setting unit 13 sets a threshold B for determining blinking based on the eye-opening distance acquired from the driver 100. In the occupant monitoring device 1, by setting the threshold B using the eye-opening distance of the driver 100, it is possible to set the threshold B to reflect the characteristics of the eye opening and closing movement of the driver 100. The setting unit 13 can set the threshold B to suit the driver 100. Furthermore, in the occupant monitoring device 1, the setting unit 13 sets a corresponding threshold B for each detection period, and the detection unit 12 detects blinking using the eye-opening distance acquired by the acquisition unit 11 and the threshold B set by the setting unit 13.

[0019] Specifically, the setting unit 13 sets the threshold value B to the eye opening speed during blinking.

[0020] The eye opening speed during a blink represents the speed at which the eyelids open during a blink. The eye opening speed during a blink is the speed at which the eyelids move from a closed state back to an open state during a blink. This eye opening speed is calculated by the detection unit 12. The detection unit 12 calculates the eye opening speed v during a blink for each detection period. The eye opening speed v during a blink is calculated using the amount of change D in the eye opening distance within the detection period and the blink opening time td using the following formula (1): v=D / td (1)

[0021] The change amount D of the eye opening distance during the detection period is the difference between the maximum and minimum values ​​of the eye opening distance during the detection period. As shown in Figure 4, the change amount D of the eye opening distance during the detection period is the difference between the maximum value EO max Minimum eye opening distance EO min 5, the detection unit 12 calculates the amount of change D in the eye opening distance during each detection period.

[0022] 5, the detection unit 12 calculates a change amount D1 in the eye-open distance during a first detection period (from time t0 to time t1). The detection unit 12 calculates a change amount D2 in the eye-open distance during a second detection period (from time t1 to time t2), a change amount D3 in the eye-open distance during a third detection period (from time t2 to time t3), a change amount D4 in the eye-open distance during a fourth detection period (from time t3 to time t4), and a change amount D5 in the eye-open distance during a fifth detection period (from time t4 to time t5). Each of the first to fifth detection periods is set to three seconds.

[0023] The blinking opening time td is the time it takes for the eyelid to change from a closed state to an open state during one blink. During a blink, the eyelids move from an open state to a closed state, and then return from the closed state to the open state. As shown in FIG. 6, the blinking opening time td is the time it takes for the eyelids to change from a closed state to an open state during one blink. The blinking opening time td is calculated by the detection unit 12. The blinking opening time td is calculated using the blink duration ta, the eye closing time tc, and the number of blinks using the following formula (2): td = (ta - tc) / (number of blinks × 2) (2)

[0024] The blink duration ta is the time taken for the eye to change from an open eye state to a closed eye state and back to an open eye state during a blink. The eye-closing time tc is the time the eye is closed during a blink. As shown in FIG. 10, the blink duration ta is expressed as the sum of the closing time tb, the closing time tc, and the opening time td during one blink. The closing time tb during a blink is the time taken for the eye to change from an open eye state to a closed eye state during one blink. The calculation of the blink duration ta, the closing time tb, and the closing time tc during a blink is performed by the detection unit 12. For example, the detection unit 12 calculates these times on a frame-by-frame basis. In the example shown in FIG. 10, the n-2th frame f n-2 From the n+4th frame f n+4 The blink duration ta is the time between the (n-2)th frame f n-2 From the nth frame f n The time tb is the time when the gap between the nth frame and the n From the n+1th frame f n+1 The eye-closing time tc is the time until the (n+1)th frame f n+1 From the n+4th frame f n+4 The opening time td is the time between the opening time td and the closing time tb. If we assume that the opening time td and the closing time tb are equal, then the above formula (2) can be expressed.

[0025] The threshold value B is set for each detection period. Also, the amount of change D in the eye-open distance during the detection period, the blinking time td, and the blinking eye-opening speed v are calculated for each detection period.

[0026] As shown in FIG. 8, the detection unit 12 calculates the amount of change D1 in the eye-open distance, the blinking time td1, and the eye-opening speed v1 during a blink during the first detection period, and the setting unit 13 sets the eye-opening speed v1 during a blink during the first detection period as the threshold value B2 for determining a blink during the second detection period. From the second detection period onward, the setting unit 13 sets the eye-opening speed v calculated during the previous detection period as the threshold value B. Furthermore, since the first detection period is the first detection period, the setting unit 13 sets an initial value for the threshold value B1 for determining a blink during the first detection period. The initial value of threshold value B is set to, for example, 75 milliseconds. The average speed of a single blink is approximately 150 milliseconds. Half of this average value is 75 milliseconds. Therefore, the initial value of threshold value B, which is the eye-opening speed during a blink, is set to 75 milliseconds.

[0027] Furthermore, the detection unit 12 performs labeling according to the comparison result between the threshold value B and the amount of change d in the eye-opening distance per unit time. As shown in Figures 9 and 10, the detection unit 12 assigns a label to each frame according to the comparison result between the threshold value B and the amount of change d in the eye-opening distance per unit time, and determines whether or not a blink has occurred based on the result of assigning the label. The detection unit 12 can assign labels to frames.

[0028] When labeling the n-th frame, the detection unit 12 performs labeling on the n-th frame f n Eye opening distance EO n From the n-1th frame f n-1 Eye opening distance EO n-1 The change in eye opening distance per unit time d is calculated by subtracting . n -EO n-1 ) is compared with threshold B.

[0029] As shown in Figure 9, n -EO n-1 >B", the label L of the nth frame n In this case, the nth frame is labeled as "1". n -EO n-1 <-B", the label L of the nth frame nIn this case, the nth frame is labeled with "-1". n -EO n-1 If EO is <|B|, a labeling condition is added and labeling is performed. n -EO n-1 <|B|" and the previous label L n-1 If is "-1", the label L of the nth frame n In this case, the nth frame is labeled as "-1". n -EO n-1 <|B|" and the previous label L n-1 If is "other than -1", the label L of the nth frame n becomes "0". In this case, the nth frame is labeled as "0".

[0030] The labeling criteria are: n -EO n-1 >B", "EO n -EO n-1 <-B", "EO n -EO n-1 < |B| and the previous label L n-1 "is -1", "EO n -EO n-1 < |B| and the previous label L n-1The label L is represented by one of four types: "1", "-1", and "0". If the amount of change d per unit time in the eye-open distance is greater than a threshold B, the detection unit 12 assigns a label L of "1", which is a first value, to the frame. If the amount of change d per unit time in the eye-open distance is less than the negative value of the threshold B, the detection unit 12 assigns a label L of "-1", which is a second value, to the frame. If the amount of change d per unit time in the eye-open distance is less than the absolute value of the threshold B and the previous label is the second value (-1), the detection unit 12 assigns a label L of "-1", which is a second value, to the frame. If the amount of change d per unit time in the eye-open distance is less than the absolute value of the threshold B and the previous label is other than the second value (other than -1), the detection unit 12 assigns a label L of "0", which is a third value, to the frame. An example of labeling performed for each frame in this manner is shown in FIG.

[0031] A method for detecting blinks based on labels will be described with reference to Fig. 10. In the example shown in Fig. 10, the (n-2)th frame f n-2 The third value, label L, of "0" is assigned to the n-1th frame f n-1 The second value, label L, of "-1" is assigned to the n-th frame f n The second value, label L, of "-1" is assigned to the n+1th frame f n+1 The second value, label L, of "-1" is assigned to the n+2th frame f n+2 The first value, label L, of "1" is assigned to the n+3th frame f n+3 The first value, label L, of "1" is assigned to the n+4th frame f n+4 The first value, label L, of "1" is assigned to the n+5th frame f n+5 A label L of "0", which is a third value, is assigned to the frame f. The detection unit 12 determines that a blink occurs between the start point and the end point, where the label changes from "0" to "-1" as a starting point and the end point as a ending point. In the example shown in FIG. 10, the timing when the label changes from "0" to "-1" is the n-2-th frame f. n-2 From the n-1th frame f n-1This timing is considered to be the start of a blink. The timing when the label changes from "1" to "0" is the n+4th frame f n+4 From the n+5th frame f n+5 This timing is determined as the end point of the blink. n-2 From the n+4th frame f n+4 The range including is determined to be one blink.

[0032] The blink detection process by the occupant monitoring device 1 will be described with reference to Fig. 11. The flowchart in Fig. 11 is executed by the occupant monitoring device 1 when the power supply of the vehicle 3 is ON, regardless of whether the vehicle 3 is moving or not.

[0033] In step S1, the face and eyes 101 of the driver 100 are detected. The acquisition unit 11 detects the face and eyes 101 of the driver 100 based on an image acquired from the camera 2. After step S1 is executed, the process proceeds to step S2.

[0034] In step S2, the eye opening distance of the eyes 101 of the driver 100 is acquired. The acquisition unit 11 calculates the eye opening distance of the eyes 101 based on the image of the eyes 101 acquired from the camera 2. After step S2 is executed, the process proceeds to step S3.

[0035] In step S3, a threshold value B for determining blinking is set. The setting unit 13 sets the threshold value B for determining blinking that corresponds to the detection period. In step S3, a subroutine shown in FIG. 12 is executed as a process for setting the threshold value for determining blinking. This subroutine will be described later with reference to FIG. 12. After step S3 is executed, the process proceeds to step S4.

[0036] In step S4, labeling is performed. The detection unit 12 performs labeling based on the eye-opening distance acquired in step S2 and the threshold B set in step S3. The detection unit 12 assigns a label to each frame according to the comparison result between the eye-opening distance and the threshold B. In step S4, a subroutine shown in FIG. 13 is executed. This subroutine will be described later with reference to FIG. 13. After step S4 is executed, the process proceeds to step S5.

[0037] In step S5, it is determined whether or not a blink has occurred. The detection unit 12 performs a blink determination process. The detection unit 12 detects a blink based on the label assigned to each frame. After step S5 has been executed, the process proceeds to step S6.

[0038] In step S6, it is determined whether the detection process within the detection period has been completed. The detection unit 12 determines whether the blink detection process for the detection period to be detected has been completed. For example, the detection unit 12 determines whether labeling of all frames within the detection period to be detected has been completed, and whether the process of determining the presence or absence of a blink based on the assigned labels has been completed. If the determination in step S6 is affirmative, the process proceeds to step S7. If the determination in step S6 is negative, the process returns to step S4.

[0039] In step S7, the amount of change D in the eye-open distance during the detection period is calculated. The detection unit 12 calculates the difference between the maximum and minimum values ​​of the eye-open distance during the detection period. For example, if the blink detection process for the first detection period has been completed and the result of the affirmative determination in step S6 is that the blink detection process for the first detection period has been completed, the amount of change D1 in the eye-open distance during the first detection period is calculated in step S7. If the blink detection process for the second detection period has been completed and the result of the affirmative determination in step S6 is that the blink detection process for the second detection period has been completed, the amount of change D2 in the eye-open distance during the second detection period is calculated in step S7. After step S7 is executed, the process proceeds to step S8.

[0040] In step S8, the blinking opening time td during the detection period is calculated. The detection unit 12 calculates the total blinking opening time td during the detection period. If a single blink is detected during the detection period, the blinking opening time td for a single blink is calculated. If multiple blinks are detected during the detection period, the total blinking opening time td for each of the multiple blinks is calculated. After step S8 is executed, the process proceeds to step S9.

[0041] In step S9, the eye opening speed v during a blink within the detection period is calculated. The detection unit 12 calculates the eye opening speed v during a blink within the detection period based on the amount of change D in the eye opening distance calculated in step S7, the blink opening time td calculated in step S8, and the number of blinks detected within the detection period. The detection unit 12 calculates the eye opening speed v during a blink using the above equations (1) and (2). After step S9 is executed, the process returns to step S3.

[0042] In this way, after a positive determination is made in step S6, the process goes through steps S7, S8, and S9 and returns to step S3, whereby blink detection processing is performed for the next detection period. The maximum eye-open distance within the detection period, the minimum eye-open distance within the detection period, the sum of the blink opening time td within the detection period, and the eye-opening speed v within the detection period can be calculated after labeling has been performed on all frames within the detection period. In short, blink information is generated for each detection period. The blink information includes the amount of change D in the eye-open distance within the detection period and the maximum eye-open distance EO within the detection period. max and the minimum eye opening distance EO within the detection period. min , the blinking opening time td, and the blinking eye opening speed v.

[0043] The blink determination threshold setting process will be described with reference to Fig. 12. As shown in Fig. 12, in the blink determination threshold setting process, in step S11, it is determined whether or not it is the first detection period. The setting unit 13 determines whether or not the detection period to be set is the first detection period. If it is the first detection period, the determination in step S11 is affirmative. If it is a detection period after the second detection period, the determination in step S11 is negative. If the determination in step S11 is affirmative, the process proceeds to step S12. If the determination in step S11 is negative, the process proceeds to step S13.

[0044] In step S12, the threshold value B for determining blinking is set to an initial value. The setting unit 13 sets the threshold value B to a predetermined initial value, for example, 75 milliseconds. When step S12 is executed, this subroutine ends and proceeds to step S4 shown in FIG. 11.

[0045] In step S13, the threshold value B for blink determination is set to the eye-opening speed v during a blink in the immediately preceding detection period. The setting unit 13 determines the threshold value B for the detection period to be set based on the eye-opening speed in the detection period immediately preceding the detection period to be set. For example, if the detection period to be set is the second detection period, the setting unit 13 sets the eye-opening speed v1 during a blink in the first detection period as the threshold value B2 for the second detection period. When step S13 is executed, this subroutine ends and the process proceeds to step S4 shown in FIG. 11.

[0046] The labeling process will be described with reference to Fig. 13. As shown in Fig. 13, in the labeling process, in step S21, the eye-opening distance within the detection period is acquired. The acquisition unit 11 acquires the eye-opening distance within the detection period that is to be detected by the detection unit 12. After step S21 is executed, the process proceeds to step S22.

[0047] In step S22, the amount of change in eye-opening distance between frames is calculated. Based on the eye-opening distance acquired in step S11, the detection unit 12 calculates the amount of change in eye-opening distance from the previous frame to the current frame within the detection period. The detection unit 12 calculates the amount of change d in the eye-opening distance per unit time. After step S22 is executed, the process proceeds to step S23.

[0048] In step S23, it is determined whether the amount of change d in the eye-opening distance between frames is greater than a threshold B for determining blinking. The detection unit 12 determines whether the amount of change d in the eye-opening distance per unit time for the current frame is greater than the threshold B. If the determination in step S23 is affirmative, the process proceeds to step S24. If the determination in step S23 is negative, the process proceeds to step S25.

[0049] In step S24, the label of the current frame is set to 1. The detection unit 12 assigns the label "1" to the current frame. When step S24 is executed, this subroutine ends, and the process proceeds to step S5 shown in FIG.

[0050] In step S25, it is determined whether the change amount d in the eye-opening distance between frames is smaller than the negative value of the threshold value B for determining blinking. The detection unit 12 determines whether the change amount d in the eye-opening distance per unit time for the current frame is smaller than the negative value of the threshold value. If the determination in step S25 is affirmative, the process proceeds to step S26. If the determination in step S25 is negative, the process proceeds to step S27.

[0051] In step S26, the label of the current frame is set to -1. The detection unit 12 assigns the label "-1" to the current frame. When step S26 is executed, this subroutine ends and the process proceeds to step S5 shown in FIG.

[0052] In step S27, it is determined whether the label assigned to the immediately previous frame is -1. The detection unit 12 determines whether the label of the frame immediately previous to the current frame is -1. If the determination in step S27 is affirmative, the process proceeds to step S28. If the determination in step S27 is negative, the process proceeds to step S29.

[0053] In step S28, the label of the current frame is set to -1. The detection unit 12 assigns the label "-1" to the current frame. When step S28 is executed, this subroutine ends and the process proceeds to step S5 shown in FIG.

[0054] In step S29, the label of the current frame is set to 0. The detection unit 12 assigns the label "0" to the current frame. When step S29 is executed, this subroutine ends and the process proceeds to step S5 shown in FIG.

[0055] The calculation process of the opening time td will be described with reference to Fig. 14. As shown in Fig. 14, in the calculation process of the opening time td, the number of blinks within the detection period is acquired in step S31. The detection unit 12 identifies the number of blinks detected within the detection period. After step S31 is executed, the process proceeds to step S32.

[0056] In step S32, it is determined whether the number of blinks is 0. The detection unit 12 determines whether the number of blinks detected within the detection period is zero. If the determination in step S32 is affirmative, the process proceeds to step S33. If the determination in step S32 is negative, the process proceeds to step S34.

[0057] In step S33, the blink opening time td is set to 75 milliseconds. If the detection unit 12 determines that a blink has not been detected within the detection period, the blink opening time td is set to 75 milliseconds. Since the average speed of a single blink is about 150 milliseconds, half of that value, 75 milliseconds, is set as the blink opening time td. When step S33 is executed, this subroutine ends and the process proceeds to step S9 shown in FIG. 11.

[0058] In step S34, the difference between the blink duration ta and the eye-closure duration tc within the detection period is calculated. The detection unit 12 calculates the value obtained by subtracting the eye-closure duration tc from the blink duration ta. After step S34 is executed, the process proceeds to step S35.

[0059] In step S35, the blink duration ta minus the eye-closing time tc is divided by twice the number of blinks to obtain a value that is set as the eye-open time td. The detection unit 12 divides the value calculated in step S34 by the number of blinks acquired in step S31, and sets half of the divided value as the eye-open time td during the detection period. When step S35 is executed, this subroutine ends, and the process proceeds to step S9 shown in FIG. 11.

[0060] The occupant monitoring device 1 configured as above improves blink detection accuracy, as shown in FIG. 15. FIG. 15(a) shows data on the eye-opening distance used to detect blinks. FIG. 15(b) shows the detection results of blinks detected using the data in FIG. 15(a) by a conventional blink detection method. FIG. 15(c) shows the detection results of blinks detected using the data in FIG. 15(a) by the blink detection method of the above-described embodiment. It can be seen that the detection method of the embodiment can detect blinks that could not be detected by conventional methods. The vertical axis in FIG. 15(a) represents the eye-opening distance, and the vertical axes in FIGS. 15(b) and 15(c) represent the blink duration (number of frames). The horizontal axes in FIGS. 15(a), 15(b), and 15(c) all represent frames.

[0061] As described above, according to the occupant monitoring device 1 of the embodiment, the threshold value B for determining blinking is set based on the eye opening distance of the driver 100, and the presence or absence of blinking is determined using the threshold value B. This makes it possible to set the threshold value B suited to the driver 100, improving the accuracy of blink detection. As a result, the occupant monitoring device 1 can accurately detect whether or not an occupant of the vehicle 3 is blinking.

[0062] Furthermore, in the occupant monitoring device 1 of the embodiment, the threshold value B for the detection period to be set is determined based on the eye-opening speed v when the eyes are opened in the detection period immediately preceding the detection period to be set. The occupant monitoring device 1 can set the threshold value B to reflect the characteristics of the eye opening and closing movement of the driver 100. This improves the accuracy of blink detection, making it possible to accurately detect whether or not an occupant of the vehicle 3 is blinking.

[0063] Furthermore, the occupant monitoring device 1 of the embodiment performs labeling according to the comparison result between the threshold value B and the amount of change d in the eye-opening distance per unit time, and determines whether the driver 100 is blinking based on the label L. The occupant monitoring device 1 can assign a label L according to the threshold value B set for the driver 100. This improves the accuracy of blink detection, making it possible to accurately detect whether the occupant of the vehicle 3 is blinking.

[0064] Furthermore, the occupant monitoring device 1 of the embodiment can determine the start and end of a blink by identifying the point where the value of the label L changes. This improves the accuracy of blink detection, and can accurately detect whether or not an occupant of the vehicle 3 is blinking.

[0065] The label L is not limited to a combination of "-1", "1", and "0". The three types of numerical values ​​used for the label L are not particularly limited. Furthermore, the label L is not limited to three types of numerical values, and may be represented by three types of symbols or alphabets.

[0066] Furthermore, the unit time is not limited to 33 milliseconds and may be set to an appropriate value. The detection period is not limited to 3 seconds and may be set to an appropriate value. Although the embodiment has been described as a detection period for the sake of convenience, this detection period can also be expressed as a unit time.

[0067] Furthermore, although the configuration for detecting blinks of the driver 100 has been described, the target is not limited to the driver 100, but can also be any occupant in the vehicle 3. For example, if two cameras 2 are installed in the driver's seat and the passenger seat, the above detection process can capture an image of the face of the passenger in the passenger seat, and detect blinks of the passenger in the passenger seat. The occupant monitoring device 1 is not limited to the driver 100, but can be applied to any occupant of the vehicle 3.

[0068] [First Modification] A first modified example of the embodiment will be described. In the occupant monitoring device 1 according to the first modified example of the embodiment, the method of setting the threshold value B is different from that of the above embodiment. Fig. 16 is a flowchart showing a threshold value setting flow in the first modified example. The flowchart shown in Fig. 16 is a modified example of the flowchart shown in Fig. 12. Steps S11, S12, and S13 shown in Fig. 16 are the same processes as steps S11, S12, and S13 shown in Fig. 12, and therefore will not be described here.

[0069] As shown in FIG. 16, in the process of setting the blink determination threshold in the first variant, if the determination in step S11 is affirmative, the process proceeds to step S12, and if the determination in step S11 is negative, the process proceeds to step S11a.

[0070] In step S11a, it is determined whether the change amount D of the eye-open distance during the detection period is greater than 0.3 times the maximum value of the eye-open distance. The setting unit 13 compares the change amount D of the eye-open distance during the detection period immediately preceding the detection period to be set with the maximum value EO of the eye-open distance. max Based on this, the amount of change D is the maximum value EO max In step S11a, the determination is made using the amount of change D in the eye-opening distance calculated in step S7 shown in Fig. 11. If the amount of change D in the eye-opening distance within the detection period is greater than or equal to 0.3 times the maximum value EO of the eye-opening distance, the determination is made. max If the change amount D of the eye opening distance during the detection period is greater than 0.3 times the maximum value EO of the eye opening distance, it can be determined that a blink has been detected during that detection period. maxIf the difference is equal to or less than 0.3 times the value of the blinking time, it can be determined that no blinking was detected during that detection period. If the answer to step S11a is affirmative, the process proceeds to step S13. If the answer to step S11a is negative, the process proceeds to step S14.

[0071] In step S13, the threshold value B for blink determination is set to the eye-opening speed v during a blink in the immediately preceding detection period. The setting unit 13 determines the threshold value B for the detection period to be set based on the eye-opening speed in the detection period immediately preceding the detection period to be set. For example, if the detection period to be set is the second detection period, the setting unit 13 sets the eye-opening speed v1 during a blink in the first detection period as the threshold value B2 for the second detection period. When step S13 is executed, this subroutine ends and the process proceeds to step S4 shown in FIG. 11.

[0072] In step S14, the maximum value EO of the eye opening distance in the detection period immediately before the detection period to be set is calculated. max by the time td during which the eyes are opened, and the calculated value is set as the threshold value B for determining the blink. The setting unit 13 sets the maximum value EO of the eye-open distance in the detection period immediately preceding the detection period to be set. max and blink opening time td, the threshold value B for the detection period to be set is determined. For example, when the detection period to be set is the second detection period, the setting unit 13 sets the maximum eye opening distance EO max The value is set as the threshold value B2 for the second detection period by dividing by the opening time td1. When step S14 is executed, this subroutine ends and the process proceeds to step S4 shown in FIG.

[0073] According to the occupant monitoring device 1 of the first modification, the threshold value B can be set in detail according to the eye opening state based on the eye opening distance of the driver 100. This improves the accuracy of blink detection.

[0074] [Second Modification] A second modified example of the embodiment will be described. Unlike the above embodiment, the occupant monitoring device 1 according to the second modified example detects microsleep of the driver 100 using the degree of eye opening. The occupant monitoring device 1 according to the second modified example executes a determination process to determine whether the driver 100 is in a drowsy state or a microsleep state. FIG. 17 is a diagram for explaining microsleep. FIG. 18 is a flowchart showing the determination process flow. FIG. 19 is a table showing the detection results in the second modified example.

[0075] As shown in Figure 17, microsleep refers to the closing of the eyes for 0.5 seconds or more but less than 3 seconds. If the eye opening degree is 100% is considered to be an awake state, the eye opening degree of 20% or less is considered to be closed. When the eye opening degree is 100%, the eyes 110 are open. When the eye opening degree is 0%, the eyes 110 are completely closed.

[0076] As shown in Fig. 18, in the discrimination process of the second modified example, in step S51, blink information obtained by the detection process of the above embodiment is acquired. This blink information includes information about the number of blinks detected within a detection period, information about frames in which labeling has been performed, and the like. The detection unit 12 acquires blink information generated in the detection period immediately preceding the detection period to be discriminated. After step S51 is performed, the process proceeds to step S52.

[0077] In step S52, the eye opening distance for 100% eye opening is acquired. The detection unit 12 acquires information about the eye opening distance set for 100% eye opening. The distance for 100% eye opening may be set by any appropriate method. After step S52 is performed, the process proceeds to step S53.

[0078] In step S53, the start of a single blink is detected. The detection unit 12 can determine the start of a blink based on the label assigned to the frame. The detection unit 12 can determine that a blink has started by detecting when the label switches from "0" to "-1". After step S53 is performed, the process proceeds to step S54.

[0079] In step S54, it is determined whether the degree of eye opening is 20% or less. The detection unit 12 determines whether the degree of eye opening of a frame determined to be a blink is 20% or less for that frame. The detection unit 12 performs the determination process for one frame at a time. If the portion determined to be a blink includes multiple frames, the detection unit 12 determines whether the degree of eye opening is 20% or less for the first frame in the blink that is labeled "-1". If the determination in step S54 is affirmative, the process proceeds to step S55. If the determination in step S54 is negative, the process proceeds to step S56.

[0080] In step S55, the counter is incremented by 1. When the eye opening degree is 20% or less, the detection unit 12 adds 1 to the counter value. After step S55 is performed, the process proceeds to step S57.

[0081] In step S56, the counter is incremented by 0. If the eye opening degree is greater than 20%, the detection unit 12 does not increment the counter. After step S56 is performed, the process proceeds to step S57.

[0082] In step S57, it is determined whether or not a single blink has ended. The detection unit 12 determines whether or not the label of the frame next to the frame determined in step S54 is "0". If the label of the next frame is "0", this is the timing when the label switches from "1" to "0", and therefore it can be determined that a single blink has ended. On the other hand, if the label of the next frame is "1" or "-1", it is determined that the blink is continuing. If the determination in step S57 is affirmative, the process proceeds to step S58. If the determination in step S57 is negative, the process returns to step S54.

[0083] The number of frames included in one blink will be determined to be negative in step S57. Therefore, the counter value is incremented by one for the number of frames in which the eye opening degree is 20% or less during one blink. The detection unit 12 determines whether the eye opening degree of the driver 100 is 20% or less for each unit time. Furthermore, the detection unit 12 counts the number of times the eye opening degree is consecutively determined to be 20% or less. Then, it is determined whether the eye opening degree is 20% or less for all frames included in one blink, the counter is incremented, and then the determination is affirmative in step S57.

[0084] In step S58, it is determined whether the counter value is 15 or greater. The detection unit 12 determines whether the counter value for a single detected blink is 15 or greater, with respect to that single blink. In this case, the detection unit 12 determines whether the eye closure time in a single blink is 0.5 seconds or greater. If the determination in step S58 is negative, the process proceeds to step S59. If the determination in step S58 is positive, the process proceeds to step S60.

[0085] In step S59, the counter value is reset to 0. If the determination in step S58 is negative, the detection unit 12 determines that the blink is a normal blink in which the eye closure time in one blink is less than 0.5 seconds, and resets the counter value to 0. After step S59 is performed, the process returns to step S53.

[0086] In step S60, it is determined whether the counter value is 90 or greater. The detection unit 12 determines whether the counter value for a single detected blink is 90 or greater, with respect to that single blink. In this case, the detection unit 12 determines whether the eyes are closed for three seconds or greater in a single blink. If the determination in step S60 is affirmative, the process proceeds to step S61. If the determination in step S60 is negative, the process proceeds to step S62.

[0087] In step S61, it is determined that the driver 100 is in a drowsy state. If the determination in step S60 is affirmative, the detection unit 12 determines that the driver is in a drowsy state in which the eyes are closed for three seconds or more in one blink. After step S61 is performed, this control routine ends.

[0088] In step S62, it is determined that the driver 100 is in a microsleep state. If the determination in step S60 is negative, the detection unit 12 determines that the driver 100 is in a microsleep state, in which the eyes are closed for 0.5 seconds or more but less than 3 seconds during a single blink. If the count is equal to or greater than the first number of 15 and less than the second number of 90, the detection unit 12 determines that the driver 100 is in a microsleep state. After step S62 is performed, this control routine ends. Note that this second modified example is not limited to the driver 100, and can also be applied to any occupant of the vehicle 3. For example, if two cameras 2 are installed, one in the driver's seat and one in the passenger's seat, microsleep can be detected for the passenger in the passenger's seat.

[0089] 19 shows the detection results of an embodiment using the detection method of the second modified example and the detection results of a comparative example using a moving average detection method. As shown in FIG. 19, the detection results of the embodiment are closer to the true value than the detection results of the comparative example. From this result, it can be seen that the second modified example makes it possible to detect blinks that cannot be detected by the moving average.

[0090] The occupant monitoring device 1 of the second modification counts the number of consecutive times that the degree of eye opening of the occupant is equal to or less than a predetermined value, and determines that the occupant is in a microsleep state when the count is equal to or greater than a first number and less than a second number. This allows microsleep in the occupant of the vehicle 3 to be detected with high accuracy.

[0091] The contents disclosed in the above-described embodiments and modifications can be implemented in appropriate combinations. [Explanation of symbols]

[0092] 1: Occupant monitoring device, 2: Camera, 3: Vehicle 11: Acquisition unit, 12: Detection unit, 13: Setting unit 100: Driver, 110: Eyes

Claims

1. an acquisition unit that acquires an eye-opening distance of a vehicle occupant; a setting unit that sets a threshold value for determining a blink based on the eye-opening distance; a detection unit that detects blinking of the occupant based on a comparison result between the threshold value and the amount of change per unit time of the eye-opening distance; Equipped with the detection unit calculates an eye opening speed of the occupant when blinking based on the eye opening distance for each predetermined detection period; the setting unit sets the threshold value corresponding to each detection period, and determines the threshold value for the detection period to be set based on the eye-opening speed in a detection period immediately preceding the detection period to be set; The detection unit A change in the eye opening distance per unit time is calculated for each unit time. assigning a label to each unit time in accordance with a comparison result between the threshold value and the amount of change in the eye-opening distance per unit time; An occupant monitoring device that determines whether the occupant is blinking based on the label assignment result.

2. The detection unit If the change amount per unit time of the eye opening distance is greater than the threshold, a first value is assigned as the label; If the change amount of the eye opening distance per unit time is smaller than a negative value of the threshold, a second value is assigned as the label; If the amount of change in the eye opening distance per unit time is smaller than the absolute value of the threshold value and if the immediately previous label is the second value, assign the second value as the label; assigning a third value as the label when the amount of change in the eye opening distance per unit time is smaller than the absolute value of the threshold value and when the immediately preceding label is a value other than the second value; A point where the label switches from the third value to the second value is set as a start point, and a point where the label switches from the first value to the third value is set as an end point, and the period from the start point to the end point is determined to be a blink. The occupant monitoring device according to claim 1 .

3. The detection unit determining for each unit time whether the degree of eye opening of the occupant during the detected blinking period of the occupant is equal to or less than a predetermined value; Counting the number of consecutive times that the eye opening degree is determined to be equal to or less than a predetermined value; If the count is equal to or greater than a first number and less than a second number, it is determined that the occupant is in a microsleep state.

3. An occupant monitoring device according to claim 1 or 2.

4. An acquisition unit that acquires the eye opening distance of a passenger in a vehicle; a setting unit that sets a threshold value for determining a blink based on the eye-opening distance; a detection unit that detects blinking of the occupant based on a comparison result between the threshold value and the amount of change per unit time of the eye-opening distance; Equipped with The detection unit determining for each unit time whether the degree of eye opening of the occupant during the detected blinking period of the occupant is equal to or less than a predetermined value; Counting the number of consecutive times that the eye opening degree is determined to be equal to or less than a predetermined value; An occupant monitoring device that determines that the occupant is in a microsleep state if the count is equal to or greater than a first number and less than a second number.

Images