Occupant monitoring device

Abstract

To provide an occupant monitoring device accurately detecting presence / absence of blinks of a vehicle occupant.SOLUTION: An occupant monitoring device 1 includes: an acquisition part 11 acquiring an eye-opening distance of an occupant on a vehicle; a detection part 12 detecting blinks of the occupant on the basis of a comparison result between a threshold value for blink determination and an amount of change in an eye-opening distance per a unit time; and a setting part 13 setting the threshold value for blink determination on the basis of the eye-open distance. The setting part 13 sets a corresponding threshold value for each of detection periods, and may decide the threshold value during the detection period of a setting target on the basis of an eye-open speed at the time of blinks during a detection period prior to the last target detection period.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an occupant monitoring device. [Background technology]

[0002] Patent Document 1 discloses an occupant monitoring device that detects whether the driver's eyes are open or closed based on image data capturing the face of the driver of a vehicle, and determines whether the driver is dozing off by using the change in eyelid opening degree over time as an indication of the eye opening or closing state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-067137 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to accurately detect whether a driver of a vehicle is dozing off while driving, it is desirable to accurately determine whether the driver's eyes are open or closed. For example, it is desirable to accurately determine whether the driver is blinking by detecting the opening and closing of the eyelids as the eye opening and closing state.

[0005] However, eyelid opening and closing movements vary from person to person, so it is difficult to accurately set a threshold value suited to an individual in a detection method that compares the degree of eye opening with a threshold value and determines that a blink has occurred when the degree of eye opening is equal to or less than the threshold value.

[0006] An object of the present invention is to provide an occupant monitoring device capable of accurately detecting whether or not a vehicle occupant is blinking. [Means for solving the problem]

[0007] The occupant monitoring device of the present invention is characterized by comprising an acquisition unit that acquires the eye-opening distance of an occupant riding in a vehicle, a setting unit that sets a threshold for determining blinking based on the eye-opening distance, and a detection unit that detects blinking of the occupant based on the comparison result between the threshold and the amount of change in the eye-opening distance per unit time. Effect of the Invention

[0008] The occupant monitoring device according to the present invention sets a threshold value for blinking determination based on the eye opening distance acquired from the vehicle occupant, so that it is possible to set a threshold value suitable for the movement of the eyelids of the occupant. Since the presence or absence of blinking is determined using this threshold value, the accuracy of blink detection is improved. As a result, the occupant monitoring device can accurately detect the presence or absence of blinking in the vehicle occupant. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a schematic diagram of a vehicle equipped with an occupant monitoring device according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the occupant monitoring device. [Diagram 3] FIG. 3 is a diagram for explaining the amount of change in eye opening distance per unit time. [Figure 4] FIG. 4 is a diagram for explaining the difference between the maximum value and the minimum value in the eye open distance within the detection period. [Diagram 5] FIG. 5 is a diagram for explaining the amount of change in eye-opening distance for each detection period. [Figure 6] FIG. 6 is a diagram for explaining the opening time during blinking. [Figure 7] FIG. 7 is a diagram for explaining a case where labeling is performed. [Figure 8] FIG. 8 is a diagram for explaining setting of a threshold value for blink determination using blink information in a detection period immediately preceding a detection target. [Figure 9] FIG. 9 is a diagram for explaining a label that is assigned according to a comparison result between a threshold value and an amount of change in eye opening distance per unit time. [Figure 10] FIG. 10 is a diagram for explaining blink duration, eye closing time, and eye opening time. [Figure 11] FIG. 11 is a flow chart showing a blink detection flow. [Figure 12] FIG. 12 is a flow chart showing a threshold setting flow. [Figure 13] FIG. 13 is a flow chart showing the labeling flow. [Figure 14] FIG. 14 is a flow chart showing the flow of calculating the opening time. [Figure 15] FIG. 15(a) is a graph showing the acquired data on eye opening distance over time, FIG. 15(b) is a graph showing blink detection results according to a comparative example, and FIG. 15(c) is a graph showing blink detection results according to an embodiment. [Figure 16] FIG. 16 is a flow chart showing a threshold setting flow in the first modified example. [Figure 17] FIG. 17 is a diagram for explaining microsleep. [Figure 18] FIG. 18 is a flow chart showing the flow of the determination process. [Figure 19] FIG. 19 is a table showing the detection results in the second modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an occupant monitoring device according to an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, the components in the following embodiment include those that can be easily imagined by a person skilled in the art or those that are substantially the same.

[0011] [Embodiment] With reference to Figs. 1 to 15, an occupant monitoring device according to an embodiment will be described. Fig. 1 is a diagram showing a vehicle equipped with the occupant monitoring device according to the embodiment. Fig. 2 is a block diagram showing a functional configuration of the occupant monitoring device. Fig. 3 is a diagram for explaining the amount of change in eye-opening distance per unit time. Fig. 4 is a diagram for explaining the difference between the maximum value and the minimum value in the eye-opening distance within a detection period. Fig. 5 is a diagram for explaining the amount of change in eye-opening distance per detection period. Fig. 6 is a diagram for explaining the opening time during blinking. Fig. 7 is a diagram for explaining a case where labeling is performed. Fig. 8 is a diagram for explaining setting a threshold value for determining a blink using blink information in a detection period immediately before the detection target. Fig. 9 is a diagram for explaining a label that is assigned according to a comparison result between the threshold value and the amount of change in eye-opening distance per unit time. Fig. 10 is a diagram for explaining the blink duration, closing time, closing time, and opening time. Fig. 11 is a flowchart showing a blink detection flow. Fig. 12 is a flowchart showing a threshold setting flow. Fig. 13 is a flowchart showing a labeling flow. Fig. 14 is a flow chart showing the calculation flow of the eye-opening time. Fig. 15(a) is a graph showing the acquired data of the eye-opening distance over time, Fig. 15(b) is a diagram showing the blink detection result by the comparative example, and Fig. 15(c) is a diagram showing the blink detection result by the embodiment.

[0012] As shown in FIG. 1, the occupant monitoring device 1 is provided in a vehicle 3 equipped with a camera 2. The occupant monitoring device 1 is a device that detects the blinking of an occupant riding in the vehicle 3. The camera 2 captures an image of the face of a driver 100 of the vehicle 3. The vehicle 3 is an automobile, and is driven by the driver 100. The camera 2 is disposed in front of the driver 100 in the cabin of the vehicle 3, and captures an image of the eyes 110 of the driver 100. The camera 2 can continuously capture an image of the face of the driver 100 at a predetermined frame rate. For example, the camera 2 can capture a video at 30 fbs.

[0013] 2, the occupant monitoring device 1 includes an acquisition unit 11, a detection unit 12, and a setting unit 13. The occupant monitoring device 1 is a computer having an arithmetic circuit, a memory, and an input / output interface. The occupant monitoring device 1 is configured as, for example, an electronic control unit (ECU).

[0014] The acquisition unit 11 acquires the eye opening distance of the driver 100 based on the images acquired from the camera 2. The eye opening distance is an index showing the open / closed state of the eyes 110, and is the distance between the upper eyelid and the lower eyelid. For example, the distance between the points where the upper eyelid and the lower eyelid are the furthest apart is the eye opening distance. The acquisition unit 11 calculates the eye opening distance of the driver 100 in each image, for example, by image recognition. In other words, the acquisition unit 11 calculates the eye opening distance for each of the images acquired continuously from the camera 2. When the eyes 110 of the driver 100 are completely closed, the eye opening distance is 0.

[0015] The acquisition unit 11 may calculate the eye opening distance for each of the left eye and the right eye of the driver 100. The eye opening distance may be an average value of the eye opening distance for the left eye and the eye opening distance for the right eye of the driver 100. The acquisition unit 11 may calculate only the eye opening distance for one of the eyes 110 of the driver 100.

[0016] The detection unit 12 detects the blinking of the driver 100 based on the eye opening distance acquired from the driver 100. The detection unit 12 calculates the amount of change in the eye opening distance per unit time within a predetermined detection period. The detection period is set to, for example, 3 seconds, that is, a length equivalent to 90 frames of image data. The unit time is set to, for example, 33 milliseconds, that is, a length between frames of image data. The detection unit 12 calculates the amount of change in the eye opening distance per unit time by subtracting the eye opening distance of the frame immediately preceding the current frame from the eye opening distance of the current frame.

[0017] As shown in FIG. 3, the detection unit 12 calculates the difference between the eye-opening distance of frame f2, which is the current frame, and the eye-opening distance of frame f1, which is the frame immediately before the current frame. This difference is the distance difference per unit time, and is the change amount d of the eye-opening distance per unit time. Then, the detection unit 12 compares the change amount d of the eye-opening distance per unit time with a threshold B for determining blinking, and determines the presence or absence of a blink based on the comparison result. In other words, the detection unit 12 does not use a threshold value of the eye-opening degree, but detects one blink by continuously determining the stage in which the upper eyelid is lowered, the stage in which the eyes are closed, and the stage in which the upper eyelid is raised based on the change in the eye-opening distance. In this description, the change amount d of the eye-opening distance per unit time may be simply referred to as the change amount d of the eye-opening distance.

[0018] The setting unit 13 sets a threshold value B for blinking determination based on the eye opening distance acquired from the driver 100. In the occupant monitoring device 1, by setting the threshold value B using the eye opening distance of the driver 100, it is possible to set the threshold value B reflecting the characteristics of the eye opening and closing movement of the driver 100. The setting unit 13 can set the threshold value B suited to the driver 100. Also, in the occupant monitoring device 1, the setting unit 13 sets a corresponding threshold value B for each detection period, and the detection unit 12 detects a blink using the eye opening distance acquired by the acquisition unit 11 and the threshold value B set by the setting unit 13.

[0019] Specifically, the setting unit 13 sets the eye opening speed during blinking to the threshold value B.

[0020] The eye opening speed during a blink represents the speed at which the eyelids open during a blink. The eye opening speed during a blink is the speed of the eyelids when returning from a closed state to an open state during a blink. This eye opening speed is calculated by the detection unit 12. The detection unit 12 calculates the eye opening speed v during a blink for each detection period. The eye opening speed v during a blink is calculated by the following formula (1) using the amount of change D in the eye opening distance within the detection period and the opening time td during a blink. v=D / td (1)

[0021] The change amount D of the eye opening distance during the detection period is the difference between the maximum and minimum values ​​of the eye opening distance during the detection period. As shown in FIG. 4, the change amount D of the eye opening distance during the detection period is the difference between the maximum value EO max Minimum eye opening distance EO min 5, the detection unit 12 calculates the amount of change D in the eye open distance during each detection period.

[0022] 5, the detection unit 12 calculates the amount of change D1 in the eye-opening distance during the first detection period (from time t0 to time t1). The detection unit 12 calculates the amount of change D2 in the eye-opening distance during the second detection period (from time t1 to time t2), the amount of change D3 in the eye-opening distance during the third detection period (from time t2 to time t3), the amount of change D4 in the eye-opening distance during the fourth detection period (from time t3 to time t4), and the amount of change D5 in the eye-opening distance during the fifth detection period (from time t4 to time t5). Each of the first to fifth detection periods is set to 3 seconds.

[0023] The blinking opening time td is the time it takes for the eyelids to change from a closed state to an open state during one blink. When blinking, the eyelids move from an open state to a closed state, and then move from the closed state back to the open state. As shown in FIG. 6, the blinking opening time td is the time it takes for the eyelids to change from a closed state to an open state during one blink. The blinking opening time td is calculated by the detection unit 12. The blinking opening time td is calculated by the following formula (2) using the blink duration ta, the eye closing time tc, and the number of blinks. td=(ta-tc) / (number of blinks x 2) (2)

[0024] The blink duration ta is the time taken for the eye to change from an open eye state to a closed eye state and back to an open eye state during a blink. The eye closed time tc is the time taken for the eye to be closed during a blink. As shown in FIG. 10, the blink duration ta is represented by the sum of the closing time tb, the closing time tc, and the opening time td during one blink. The closing time tb during a blink is the time taken for the eye to change from an open eye state to a closed eye state during one blink. The blink duration ta and the closing time tb and closing time tc during a blink are calculated by the detection unit 12. For example, the detection unit 12 calculates these times on a frame-by-frame basis. In the example shown in FIG. 10, the n-2th frame f n-2 From the n+4th frame f n+4 The blink duration ta is the time between the n-2th frame f n-2 From the nth frame f n The time tb when the gap between the nth frame and the n From the n+1th frame f n+1 The eye-closing time tc is the time until the n+1th frame f n+1 From the n+4th frame f n+4 The time it takes for the door to open is td. If we assume that the opening time td and the closing time tb are equal, then we get the above formula (2).

[0025] The threshold value B is set for each detection period. In addition, the change amount D of the eye opening distance during the detection period, the blinking opening time td, and the blinking eye opening speed v are also calculated for each detection period.

[0026] As shown in FIG. 8, the detection unit 12 calculates the change amount D1 of the eye opening distance during the first detection period, the time td1 to open the eye during a blink, and the eye opening speed v1 during a blink, and the setting unit 13 sets the eye opening speed v1 during a blink during the first detection period as the threshold value B2 for determining a blink during the second detection period. After the second detection period, the setting unit 13 sets the eye opening speed v calculated during the previous detection period as the threshold value B. In addition, since the first detection period is the first detection period, the setting unit 13 sets an initial value to the threshold value B1 for determining a blink during the first detection period. The initial value of the threshold value B is set to, for example, 75 milliseconds. The average value of the speed of one blink is about 150 milliseconds. Half of this average value is 75 milliseconds. Therefore, the initial value of the threshold value B, which is the eye opening speed during a blink, is set to 75 milliseconds.

[0027] Furthermore, the detection unit 12 performs labeling according to a comparison result between the threshold value B and the amount of change d in the eye-opening distance per unit time. As shown in Fig. 9 and Fig. 10, the detection unit 12 assigns a label according to a comparison result between the threshold value B and the amount of change d in the eye-opening distance per unit time for each unit time, and determines the presence or absence of a blink based on the result of the assignment of the label. The detection unit 12 is capable of assigning labels to frames.

[0028] When labeling the n-th frame, the detection unit 12 performs labeling on the n-th frame f as shown in FIG. n Eye opening distance EO n From the n-1th frame f n-1 Eye opening distance EO n-1 The change in the eye opening distance per unit time d is calculated by subtracting EO n -EO n-1 ) is compared with threshold B.

[0029] As shown in Figure 9, "EO n -EO n-1 >B", the label L of the nth frame n In this case, the nth frame is labeled with "1". n -EO n-1 <-B", the label L of the nth frame nIn this case, the nth frame is labeled with "-1". n -EO n-1 If <|B|, add a labeling condition and perform labeling. n -EO n-1 <|B|" and the previous label L n-1 If is "-1", the label L of the nth frame n In this case, the nth frame is labeled with "-1". n -EO n-1 <|B|" and the previous label L n-1 If is "other than -1", the label L of the nth frame n becomes "0". In this case, the nth frame is given the label "0".

[0030] The labeling criteria are: "EO n -EO n-1 >B", "EO n -EO n-1 <-B", "EO n -EO n-1 < |B| and the previous label L n-1 " is -1", "EO n -EO n-1 < |B| and the previous label L n-1The label L is represented by four types: "1", "-1", and "0". When the change amount d of the eye open distance per unit time is greater than the threshold B, the detection unit 12 assigns the label L of "1", which is a first value, to the frame. When the change amount d of the eye open distance per unit time is less than the negative value of the threshold B, the detection unit 12 assigns the label L of "-1", which is a second value, to the frame. When the change amount d of the eye open distance per unit time is less than the absolute value of the threshold B and the previous label is the second value (-1), the detection unit 12 assigns the label L of "-1", which is a second value, to the frame. When the change amount d of the eye open distance per unit time is less than the absolute value of the threshold B and the previous label is other than the second value (other than -1), the detection unit 12 assigns the label L of "0", which is a third value, to the frame. An example of frame-by-frame labeling performed in this manner is shown in FIG.

[0031] A method for detecting blinks based on labels will be described with reference to FIG. 10. In the example shown in FIG. 10, the n-2th frame f n-2 The third value, label L, of “0”, is assigned to the n-1th frame f n-1 The second value, label L, of "-1" is assigned to the nth frame f n The second value, label L, of "-1" is assigned to the n+1th frame f n+1 The second value, label L, of "-1" is assigned to the n+2th frame f n+2 The label L of the first value "1" is assigned to the n+3th frame f n+3 The label L of the first value "1" is assigned to the n+4th frame f n+4 The label L of the first value "1" is assigned to the n+5th frame f n+5 A label L of a third value "0" is assigned to each of the frames f1 and f2. The detection unit 12 determines that a blink occurs between the start point where the label switches from "0" to "-1" and the end point where the label switches from "1" to "0". In the example shown in FIG. 10, the timing when the label switches from "0" to "-1" is the n-2-th frame f n-2 From the n-1th frame f n-1This is the timing when the label switches from "1" to "0". This timing is judged as the start of a blink. The timing when the label switches from "1" to "0" is the n+4th frame f n+4 From the n+5th frame f n+5 This timing is determined as the end point of a blink. n-2 From the n+4th frame f n+4 The range including the blinking period is determined as one blinking period.

[0032] A blink detection process by the occupant monitoring device 1 will be described with reference to Fig. 11. The flowchart in Fig. 11 is executed by the occupant monitoring device 1 when the power supply of the vehicle 3 is ON, regardless of whether the vehicle 3 is running or not.

[0033] In step S1, the face and eyes 101 of the driver 100 are detected. The acquisition unit 11 detects the face and eyes 101 of the driver 100 based on an image acquired from the camera 2. After step S1 is executed, the process proceeds to step S2.

[0034] In step S2, the distance the eyes 101 of the driver 100 are opened is acquired. The acquisition unit 11 calculates the distance the eyes 101 are opened based on the image of the eyes 101 acquired from the camera 2. After step S2 is executed, the process proceeds to step S3.

[0035] In step S3, a threshold value B for blink determination is set. The setting unit 13 sets the threshold value B for blink determination corresponding to the detection period. In step S3, a subroutine shown in FIG. 12 is executed as a process for setting the threshold value for blink determination. This subroutine will be described later with reference to FIG. After step S3 is executed, the process proceeds to step S4.

[0036] In step S4, labeling is performed. The detection unit 12 performs labeling based on the eye open distance acquired in step S2 and the threshold B set in step S3. The detection unit 12 assigns a label to each frame according to the comparison result between the eye open distance and the threshold B. In step S4, a subroutine shown in Fig. 13 is executed. This subroutine will be described later with reference to Fig. 13. After step S4 is executed, the process proceeds to step S5.

[0037] In step S5, the presence or absence of a blink is determined. The detection unit 12 performs a blink determination process. The detection unit 12 detects a blink based on the label assigned to each frame. After step S5 is executed, the process proceeds to step S6.

[0038] In step S6, it is determined whether the detection process within the detection period has been completed. The detection unit 12 determines whether the blink detection process for the detection period to be detected has been completed. For example, the detection unit 12 determines whether labeling has been completed for all frames within the detection period to be detected, and whether the process of determining the presence or absence of a blink based on the assigned labels has been completed. If the determination in step S6 is affirmative, the process proceeds to step S7. If the determination in step S6 is negative, the process returns to step S4.

[0039] In step S7, the amount of change D in the eye open distance during the detection period is calculated. The detection unit 12 calculates the difference between the maximum and minimum values ​​of the eye open distance during the detection period. For example, if the blink detection process for the first detection period is completed and the result is a positive determination in step S6, the amount of change D1 in the eye open distance during the first detection period is calculated in step S7. If the blink detection process for the second detection period is completed and the result is a positive determination in step S6, the amount of change D2 in the eye open distance during the second detection period is calculated in step S7. Once step S7 has been executed, the process proceeds to step S8.

[0040] In step S8, the blinking opening time td during the detection period is calculated. The detection unit 12 calculates the total blinking opening time td during the detection period. If one blink is detected during the detection period, the blinking opening time td for one blink is calculated. If multiple blinks are detected during the detection period, the total of the blinking opening times td for multiple blinks is calculated. After step S8 is executed, the process proceeds to step S9.

[0041] In step S9, the eye opening speed v during a blink within the detection period is calculated. The detection unit 12 calculates the eye opening speed v during a blink within the detection period based on the amount of change D in the eye opening distance calculated in step S7, the blink opening time td calculated in step S8, and the number of blinks detected within the detection period. The detection unit 12 calculates the eye opening speed v during a blink using the above formulas (1) and (2). After step S9 is executed, the process returns to step S3.

[0042] In this way, after a positive determination is made in step S6, the process goes through steps S7, S8, and S9, and then returns to step S3, whereby blink detection processing is performed with the next detection period as the detection target. The maximum eye open distance within the detection period, the minimum eye open distance within the detection period, the sum of the blink opening time td within the detection period, and the eye open speed v during blinking within the detection period can be calculated after labeling has been performed on all frames within the detection period. In short, blink information is generated for each detection period. The blink information includes the amount of change D in the eye open distance within the detection period, the maximum eye open distance EO within the detection period, and the maximum value of the eye open distance EO during the detection period. max and the minimum eye opening distance EO during the detection period. min , the blinking opening time td, and the blinking eye opening speed v.

[0043] The process of setting the blink determination threshold will be described with reference to Fig. 12. As shown in Fig. 12, in the process of setting the blink determination threshold, in step S11, it is determined whether or not it is the first detection period. The setting unit 13 determines whether or not the detection period to be set is the first detection period. If it is the first detection period, it is determined affirmatively in step S11. If it is a second detection period or a subsequent detection period, it is determined negatively in step S11. If the determination in step S11 is affirmative, the process proceeds to step S12. If the determination in step S11 is negative, the process proceeds to step S13.

[0044] In step S12, the threshold value B for determining a blink is set to an initial value. The setting unit 13 sets the threshold value B to a predetermined initial value, for example, 75 milliseconds. When step S12 is executed, this subroutine ends and proceeds to step S4 shown in FIG.

[0045] In step S13, the threshold value B for blink determination is set to the eye opening speed v at the time of blinking in the previous detection period. The setting unit 13 determines the threshold value B for the detection period to be set based on the eye opening speed in the detection period immediately preceding the detection period to be set. For example, when the detection period to be set is the second detection period, the setting unit 13 sets the eye opening speed v1 at the time of blinking in the first detection period to the threshold value B2 for the second detection period. When step S13 is executed, this subroutine ends and proceeds to step S4 shown in FIG. 11.

[0046] The labeling process will be described with reference to Fig. 13. As shown in Fig. 13, in the labeling process, in step S21, the eye open distance within the detection period is acquired. The acquisition unit 11 acquires the eye open distance within the detection period to be detected by the detection unit 12. After step S21 is executed, the process proceeds to step S22.

[0047] In step S22, the amount of change in eye open distance between frames is calculated. Based on the eye open distance acquired in step S11, the detection unit 12 calculates the amount of change in eye open distance from the previous frame to the current frame within the detection period. The detection unit 12 calculates the amount of change d of the eye open distance per unit time. After step S22 is executed, the process proceeds to step S23.

[0048] In step S23, it is determined whether the amount of change d in the eye open distance between frames is greater than the threshold B for determining blinking. The detection unit 12 determines whether the amount of change d in the eye open distance per unit time is greater than the threshold B for the current frame to be added. If the determination in step S23 is affirmative, the process proceeds to step S24. If the determination in step S23 is negative, the process proceeds to step S25.

[0049] In step S24, the label of the current frame is set to 1. The detection unit 12 assigns the label "1" to the current frame. When step S24 is executed, this subroutine ends, and the process proceeds to step S5 shown in FIG.

[0050] In step S25, it is determined whether the change amount d of the eye open distance between frames is smaller than the negative value of the threshold value B for determining blinking. The detection unit 12 determines whether the change amount d of the eye open distance per unit time is smaller than the negative value of the threshold value for the current frame to be added. If the determination in step S25 is affirmative, the process proceeds to step S26. If the determination in step S25 is negative, the process proceeds to step S27.

[0051] In step S26, the label of the current frame is set to -1. The detection unit 12 assigns the label "-1" to the current frame. When step S26 is executed, this subroutine ends, and the process proceeds to step S5 shown in FIG.

[0052] In step S27, it is determined whether or not the label assigned to the previous frame is -1. The detection unit 12 determines whether or not the label of the frame immediately prior to the current frame is -1. If the determination in step S27 is affirmative, the process proceeds to step S28. If the determination in step S27 is negative, the process proceeds to step S29.

[0053] In step S28, the label of the current frame is set to -1. The detection unit 12 assigns the label "-1" to the current frame. When step S28 is executed, this subroutine ends, and the process proceeds to step S5 shown in FIG.

[0054] In step S29, the label of the current frame is set to 0. The detection unit 12 assigns the label "0" to the current frame. When step S29 is executed, this subroutine ends, and the process proceeds to step S5 shown in FIG.

[0055] The calculation process of the opening time td will be described with reference to Fig. 14. As shown in Fig. 14, in the calculation process of the opening time td, in step S31, the number of blinks within the detection period is acquired. The detection unit 12 specifies the number of blinks detected within the detection period. After step S31 is executed, the process proceeds to step S32.

[0056] In step S32, it is determined whether the number of blinks is 0. The detection unit 12 determines whether the number of blinks detected within the detection period is zero. If the determination in step S32 is affirmative, the process proceeds to step S33. If the determination in step S32 is negative, the process proceeds to step S34.

[0057] In step S33, the blinking opening time td is set to 75 milliseconds. When the detection unit 12 determines that a blink has not been detected within the detection period, the blinking opening time td is set to 75 milliseconds. Since the average blink speed is about 150 milliseconds, half of that value, 75 milliseconds, is set as the blinking opening time td. When step S33 is executed, this subroutine ends and the process proceeds to step S9 shown in FIG. 11.

[0058] In step S34, the difference between the blink duration ta and the eye-closing duration tc within the detection period is calculated. The detection unit 12 calculates a value by subtracting the eye-closing duration tc from the blink duration ta. After step S34 is executed, the process proceeds to step S35.

[0059] In step S35, the blink duration ta minus the eye closing time tc is divided by twice the blink count, and this value is set as the eye opening time td. The detection unit 12 divides the value calculated in step S34 by the blink count acquired in step S31, and sets half of this divided value as the eye opening time td during the detection period. When step S35 is executed, this subroutine ends, and the process proceeds to step S9 shown in FIG.

[0060] In the passenger monitoring device 1 configured in this manner, the blink detection accuracy is improved as shown in FIG. 15. FIG. 15(a) shows data on the eye opening distance used to detect the blink. FIG. 15(b) shows the detection result of the conventional blink detection method using the data in FIG. 15(a). FIG. 15(c) shows the detection result of the conventional blink detection method using the data in FIG. 15(a). It can be seen that the detection method of the embodiment can detect blinks that could not be detected by the conventional method. The vertical axis of FIG. 15(a) shows the eye opening distance, and the vertical axes of FIG. 15(b) and FIG. 15(c) show the blink duration (number of frames). The horizontal axes of FIG. 15(a), FIG. 15(b), and FIG. 15(c) all show frames.

[0061] As described above, according to the occupant monitoring device 1 of the embodiment, the threshold value B for determining blinking is set based on the eye opening distance of the driver 100, and the presence or absence of blinking is determined using the threshold value B. This makes it possible to set the threshold value B suited to the driver 100, improving the accuracy of blink detection. As a result, the occupant monitoring device 1 can accurately detect the presence or absence of blinking of an occupant of the vehicle 3.

[0062] Furthermore, in the occupant monitoring device 1 of the embodiment, the threshold value B of the detection period to be set is determined based on the eye-opening speed v when the eyes are opened in the detection period immediately preceding the detection period to be set. The occupant monitoring device 1 can set the threshold value B reflecting the characteristics of the eye opening and closing movement of the driver 100. This improves the accuracy of blink detection, and makes it possible to accurately detect whether or not the occupant of the vehicle 3 is blinking.

[0063] Furthermore, in the occupant monitoring device 1 of the embodiment, labeling is performed according to a comparison result between the threshold value B and the amount of change d in the eye-opening distance per unit time, and the presence or absence of the blinking of the driver 100 is determined based on the label L. The occupant monitoring device 1 can assign the label L according to the threshold value B set for the driver 100. This improves the accuracy of blink detection, and makes it possible to accurately detect the presence or absence of the blinking of the occupant of the vehicle 3.

[0064] In addition, the occupant monitoring device 1 of the embodiment can determine the start and end of a blink by identifying the point where the value of the label L changes. This improves the accuracy of blink detection, and can accurately detect whether or not an occupant of the vehicle 3 is blinking.

[0065] The label L is not limited to a combination of "-1", "1" and "0". The three types of numerical values ​​used for the label L are not particularly limited. The label L is not limited to three types of numerical values, and may be represented by three types of symbols or English letters.

[0066] In addition, the unit time is not limited to 33 milliseconds and may be set to an appropriate value. The detection period is not limited to 3 seconds and may be set to an appropriate value. Although the detection period is described in the embodiment for convenience of explanation, this detection period can be expressed as a unit time.

[0067] Also, although the configuration for detecting blinking of the driver 100 has been described, it is possible to target not only the driver 100 but also any passenger in the vehicle 3. For example, when two cameras 2 are installed in the driver's seat and the passenger seat, it is possible to capture an image of the face of the passenger in the passenger seat by the above detection process and detect blinking of the passenger in the passenger seat. The passenger monitoring device 1 is not limited to the driver 100 but can be applied to any passenger in the vehicle 3.

[0068] [First Modification] A first modified example of the embodiment will be described. In the occupant monitoring device 1 according to the first modified example of the embodiment, a method for setting the threshold value B is different from that of the above embodiment. FIG. 16 is a flowchart showing a threshold value setting flow in the first modified example. The flowchart shown in FIG. 16 is a modified example of the flowchart shown in FIG. 12. Steps S11, S12, and S13 shown in FIG. 16 are similar to steps S11, S12, and S13 shown in FIG. 12, and therefore will not be described here.

[0069] As shown in FIG. 16, in the process of setting a threshold value for blink determination in the first modified example, if a positive determination is made in step S11, the process proceeds to step S12, and if a negative determination is made in step S11, the process proceeds to step S11a.

[0070] In step S11a, it is determined whether the change amount D of the eye open distance during the detection period is greater than 0.3 times the maximum value of the eye open distance. The setting unit 13 determines whether the change amount D of the eye open distance during the detection period immediately preceding the detection period to be set and the maximum value EO of the eye open distance. max Based on this, the amount of change D is the maximum value EO max In step S11a, the determination is made using the amount of change D in the eye-opening distance calculated in step S7 shown in Fig. 11. The amount of change D in the eye-opening distance during the detection period is determined to be greater than or equal to 0.3 times the maximum value EO of the eye-opening distance. max If the change amount D of the eye opening distance during the detection period is greater than 0.3 times the maximum value EO of the eye opening distance, it can be determined that a blink has been detected during that detection period. maxIf the difference is equal to or less than 0.3 times the value of the detection period, it can be determined that no blinking has been detected during that detection period. If the determination in step S11a is affirmative, the process proceeds to step S13. If the determination in step S11 is negative, the process proceeds to step S14.

[0071] In step S13, the threshold value B for blink determination is set to the eye opening speed v at the time of blinking in the previous detection period. The setting unit 13 determines the threshold value B for the detection period to be set based on the eye opening speed in the detection period immediately preceding the detection period to be set. For example, when the detection period to be set is the second detection period, the setting unit 13 sets the eye opening speed v1 at the time of blinking in the first detection period to the threshold value B2 for the second detection period. When step S13 is executed, this subroutine ends and proceeds to step S4 shown in FIG. 11.

[0072] In step S14, the maximum eye opening distance EO in the detection period immediately preceding the detection period to be set is set. max The value is calculated by dividing the maximum eye-opening distance EO by the time td for which the eye is opened, and the calculated value is set as the threshold value B for blinking determination. max For example, when the detection period to be set is the second detection period, the setting unit 13 sets the maximum eye opening distance EO max by the opening time td1, and the value is set as the threshold value B2 for the second detection period. When step S14 is executed, this subroutine ends, and the process proceeds to step S4 shown in FIG.

[0073] According to the occupant monitoring device 1 of the first modified example, it is possible to set the threshold value B in detail according to the eye opening state based on the eye opening distance of the driver 100. This improves the blink detection accuracy.

[0074] [Second modified example] A second modified example of the embodiment will be described. Unlike the above embodiment, the occupant monitoring device 1 according to the second modified example detects microsleep of the driver 100 using the degree of eye opening. The occupant monitoring device 1 according to the second modified example executes a determination process to determine whether the driver 100 is in a drowsy state or in a microsleep state. FIG. 17 is a diagram for explaining microsleep. FIG. 18 is a flow chart showing a determination process flow. FIG. 19 is a table showing the detection results in the second modified example.

[0075] As shown in Fig. 17, microsleep refers to the closing of the eyes for 0.5 seconds or more and less than 3 seconds. If the eye opening degree of 100% is defined as an awake state, the eye opening degree of 20% or less is defined as closed. When the eye opening degree is 100%, the eyes 110 are open. When the eye opening degree is 0%, the eyes 110 are completely closed.

[0076] As shown in Fig. 18, in the discrimination process in the second modified example, in step S51, blink information in which the detection process of the above embodiment has been performed is acquired. This blink information includes information on the number of blinks detected within the detection period, and information on the frame in which labeling has been performed. The detection unit 12 acquires blink information generated in the detection period immediately preceding the detection period to be discriminated. After step S51 has been performed, the process proceeds to step S52.

[0077] In step S52, the eye opening distance for 100% eye opening is acquired. The detection unit 12 acquires information about the eye opening distance set for 100% eye opening. The distance for 100% eye opening may be set by any appropriate method. After step S52 is performed, the process proceeds to step S53.

[0078] In step S53, the start of a single blink is detected. The detection unit 12 can determine the start of a blink based on the label attached to the frame. The detection unit 12 can determine that a blink has started by detecting where the label switches from "0" to "-1". After step S53 is performed, the process proceeds to step S54.

[0079] In step S54, it is determined whether the degree of eye opening is 20% or less. The detection unit 12 determines whether the degree of eye opening of a frame determined to be a blink is 20% or less for that frame. This detection unit 12 performs the determination process for one frame. If the portion determined to be a blink includes multiple frames, the detection unit 12 determines whether the degree of eye opening is 20% or less for the first frame labeled "-1" in one blink. If the determination in step S54 is affirmative, the process proceeds to step S55. If the determination in step S54 is negative, the process proceeds to step S56.

[0080] In step S55, the counter is incremented by 1. When the eye opening degree is 20% or less, the detection unit 12 adds 1 to the counter value. After step S55 is performed, the process proceeds to step S57.

[0081] In step S56, the counter is incremented by 0. If the eye opening degree is greater than 20%, the detection unit 12 does not increment the counter. After step S56 is performed, the process proceeds to step S57.

[0082] In step S57, it is determined whether or not a single blink has ended. The detection unit 12 determines whether or not the label of the frame next to the frame determined in step S54 is "0". If the label of the next frame is "0", this is the timing when the label switches from "1" to "0", and therefore it can be determined that a single blink has ended. On the other hand, if the label of the next frame is "1" or "-1", it is determined that the blink is continuing. If the determination in step S57 is affirmative, the process proceeds to step S58. If the determination in step S57 is negative, the process returns to step S54.

[0083] The number of frames included in one blink will be determined to be negative in step S57. Therefore, the counter value is incremented by one for the number of frames in which the eye opening degree is 20% or less during one blink. The detection unit 12 determines whether the eye opening degree of the driver 100 is 20% or less for each unit time. Furthermore, the detection unit 12 counts the number of times that the eye opening degree is consecutively determined to be 20% or less. Then, it is determined whether the eye opening degree is 20% or less for all frames included in one blink, and after the counter is incremented, the determination is affirmative in step S57.

[0084] In step S58, it is determined whether the counter value is 15 or more. The detection unit 12 determines whether the counter value in a detected blink is 15 or more for a single blink. In this case, the detection unit 12 determines whether the eye closure in a single blink lasts for 0.5 seconds or more. If the determination in step S58 is negative, the process proceeds to step S59. If the determination in step S58 is positive, the process proceeds to step S60.

[0085] In step S59, the counter value is reset to 0. If the determination in step S58 is negative, the detection unit 12 determines that the blink is a normal blink in which the eye closure period in one blink is less than 0.5 seconds, and resets the counter value to 0. After step S59 is performed, the process returns to step S53.

[0086] In step S60, it is determined whether the counter value is 90 or more. The detection unit 12 determines whether the counter value for a single blink detected is 90 or more. In this case, the detection unit 12 determines whether the eyes are closed for 3 seconds or more in a single blink. If the determination in step S60 is positive, the process proceeds to step S61. If the determination in step S60 is negative, the process proceeds to step S62.

[0087] In step S61, it is determined that the driver 100 is in a drowsy state. If the determination in step S60 is affirmative, the detection unit 12 determines that the driver is in a drowsy state in which the eyes are closed for three seconds or more in one blink. After step S61 is performed, this control routine ends.

[0088] In step S62, it is determined that the driver 100 is in a microsleep state. If the determination in step S60 is negative, the detection unit 12 determines that the driver 100 is in a microsleep state in which the eyes are closed for 0.5 seconds or more and less than 3 seconds in one blink. If the count is equal to or more than the first number of 15 and less than the second number of 90, the detection unit 12 determines that the driver 100 is in a microsleep state. When step S62 is performed, this control routine ends. Note that this second modified example is not limited to the driver 100, and can be applied to any occupant of the vehicle 3. For example, when two cameras 2 are installed in the driver's seat and the passenger's seat, it is possible to detect microsleep for the passenger in the passenger's seat.

[0089] Fig. 19 shows the detection result of the embodiment using the detection method of the second modification and the detection result of the comparative example using the moving average detection method. As shown in Fig. 19, the detection result of the embodiment is closer to the true value than the detection result of the comparative example. From this result, it can be seen that the second modification can detect blinks that cannot be detected by the moving average.

[0090] According to the occupant monitoring device 1 of the second modification, the number of consecutive times that the degree of eye opening of the occupant is equal to or less than a predetermined value is counted, and when the count is equal to or greater than a first number and less than a second number, it is determined that the occupant is in a microsleep state. This makes it possible to accurately detect microsleep in an occupant of the vehicle 3.

[0091] The contents disclosed in the above-described embodiment and modified examples can be implemented in appropriate combinations. [Explanation of symbols]

[0092] 1: Occupant monitoring device; 2: Camera; 3: Vehicle 11: Acquisition unit, 12: Detection unit, 13: Setting unit 100: Driver, 110: Eyes

Claims

1. An acquisition unit that acquires an eye opening distance of a passenger in a vehicle; A setting unit that sets a threshold value for determining a blink based on the eye opening distance; a detection unit that detects the blinking of the occupant based on a comparison result between the threshold value and an amount of change in the eye opening distance per unit time; An occupant monitoring device comprising:

2. The detection unit calculates an eye opening speed at the time of blinking of the occupant based on the eye opening distance for each predetermined detection period, The setting unit sets the threshold value corresponding to each detection period, and determines the threshold value for the detection period to be set based on the eye-opening speed in a detection period immediately preceding the detection period to be set.

2. An occupant monitoring device according to claim 1.

3. The detection unit is A change in the eye opening distance per unit time is calculated for each unit time. assigning a label to each unit time in accordance with a comparison result between the threshold value and the amount of change in the eye opening distance per unit time; Based on the label assignment result, it is determined whether the occupant blinks or not.

3. An occupant monitoring device according to claim 2.

4. The detection unit is When the amount of change in the eye opening distance per unit time is greater than the threshold, a first value is assigned as the label; and When the amount of change in the eye opening distance per unit time is smaller than a negative value of the threshold, a second value is assigned as the label; When the amount of change in the eye opening distance per unit time is smaller than the absolute value of the threshold and when the immediately previous label is the second value, the second value is assigned as the label; when the amount of change in the eye opening distance per unit time is smaller than the absolute value of the threshold and when the immediately preceding label is a value other than the second value, a third value is assigned as the label; A point where the label switches from the third value to the second value is set as a start point, and a point where the label switches from the first value to the third value is set as an end point, and a period from the start point to the end point is determined to be a blink.

4. An occupant monitoring device according to claim 3.

5. The detection unit is determining whether the degree of eye opening of the occupant is equal to or less than a predetermined value for each unit time; Counting the number of consecutive times that the eye opening degree is determined to be equal to or less than a predetermined value; If the count is greater than or equal to a first number and less than a second number, the occupant is determined to be in a microsleep state. An occupant monitoring device according to any one of claims 1 to 4.

Images