Crew member monitoring device

Abstract

To provide a crew member monitoring device capable of accurately detecting the abnormal state of a vehicle crew member.SOLUTION: A crew member monitoring device 1 includes a calculation unit 12 for calculating the eye opening degree of a crew member aboard a vehicle and a determination unit 13 for determining the state of the crew member on the basis of the eye closing duration time of the crew member and the minimum eye opening degree in the eye closing duration time. The calculation unit 12 excludes a time corresponding to the state in which the eye closing duration time is a prescribed time or more and the minimum eye opening degree is larger than a prescribed value of the eye closing duration time from the eye closing time used for the calculation of PERCLOS. The determination unit 13 may determine the drowsiness of the crew member on the basis of the PERCLOS indicating the ratio of the eye closing time per unit time.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an occupant monitoring device. [Background technology]

[0002] Patent Document 1 discloses a device that acquires the degree of eye opening of a subject, calculates the variation in the degree of eye opening within a predetermined time, and performs a drowsiness determination using the calculated variation and a predetermined threshold value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-130342 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to accurately detect whether a driver is dozing while driving a vehicle, it is desirable to accurately determine whether the driver's eyes are open or closed. For example, it is possible to determine the driver's drowsiness based on PERCLOS, which indicates the ratio of time the eyes are closed per unit time. PERCLOS is calculated by acquiring the time the driver's eyes are closed.

[0005] However, in reality, there are blinks that are unrelated to drowsiness, such as squinting the eyes due to the glare of sunlight or smiling, and the closed eye time due to this unrelated blink is acquired. Therefore, the park crossing is calculated including the unrelated closed eye time, and when detecting an abnormal state of the driver, such as drowsiness, based on this park crossing, the detection accuracy is reduced.

[0006] An object of the present invention is to provide an occupant monitoring device capable of detecting an abnormal state of a vehicle occupant with high accuracy. [Means for solving the problem]

[0007] The occupant monitoring device of the present invention is characterized by comprising a calculation unit that calculates the degree of eye opening of an occupant riding in a vehicle, and a judgment unit that judges the state of the occupant based on the duration of the occupant's eyes being closed and the minimum degree of eye opening within the duration of the eyes being closed. Effect of the Invention

[0008] The occupant monitoring device according to the present invention can accurately detect an abnormal state of a vehicle occupant. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a schematic diagram of a vehicle equipped with an occupant monitoring device according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the occupant monitoring device. [Diagram 3] FIG. 3 is a diagram for explaining PERCLOS. [Figure 4] FIG. 4 is a diagram for explaining four types of eye opening and closing movements. [Diagram 5] FIG. 5 is a diagram for explaining the duration of eye closure during one blink. [Figure 6] FIG. 6 is a diagram showing the proportion of types among people who blink deeply a lot. [Figure 7] FIG. 7 is a diagram showing the proportion of types in people who blink a lot shallowly. [Figure 8] FIG. 8 is a flow chart showing a flow of setting the drowsiness determination criteria. [Figure 9] FIG. 9 is a flowchart showing the blink rate calculation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an occupant monitoring device according to an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, the components in the following embodiment include those that can be easily imagined by a person skilled in the art or those that are substantially the same.

[0011] An occupant monitoring device according to an embodiment will be described with reference to Figs. 1 to 9. Fig. 1 is a diagram showing a model of a vehicle equipped with an occupant monitoring device according to an embodiment. Fig. 2 is a block diagram showing a functional configuration of the occupant monitoring device. Fig. 3 is a diagram for explaining PERCLOS. Fig. 4 is a diagram for explaining four types of eye opening and closing actions. Fig. 5 is a diagram for explaining the duration of eye closure in one blink. Fig. 6 is a diagram showing the proportion of types in people who blink deeply a lot. Fig. 7 is a diagram showing the proportion of types in people who blink shallowly a lot. Fig. 8 is a flow chart showing a flow of setting a drowsiness determination criterion. Fig. 9 is a flow chart showing a process of calculating a blink rate.

[0012] As shown in FIG. 1, the occupant monitoring device 1 is provided in a vehicle 3 equipped with a camera 2. The occupant monitoring device 1 is a device that detects the blinking of an occupant riding in the vehicle 3. The camera 2 captures an image of the face of a driver 4 of the vehicle 3. The vehicle 3 is an automobile, and is driven by the driver 4. The camera 2 is disposed in front of the driver 4 in the cabin of the vehicle 3, and captures an image of the eyes 5 of the driver 4. The camera 2 can continuously capture an image of the face of the driver 4 at a predetermined frame rate. For example, the camera 2 can capture a video at 30 fbs.

[0013] 2, the occupant monitoring device 1 includes an acquisition unit 11, a calculation unit 12, and a determination unit 13. The occupant monitoring device 1 is a computer having an arithmetic circuit, a memory, and an input / output interface. The occupant monitoring device 1 is configured as, for example, an electronic control unit (ECU).

[0014] The acquisition unit 11 acquires the eye opening distance of the driver 4 based on the images acquired from the camera 2. The eye opening distance is an index showing the open / closed state of the eyes 5, and is the distance between the upper eyelid and the lower eyelid. For example, the distance at the point where the upper eyelid and the lower eyelid are farthest apart is the eye opening distance. The acquisition unit 11 calculates the eye opening distance of the driver 4 in each image, for example, by image recognition. The acquisition unit 11 calculates the eye opening distance for each of the images acquired continuously from the camera 2. When the eyes 5 of the driver 4 are completely closed, the eye opening distance is 0.

[0015] The calculation unit 12 calculates the degree of eye opening of the driver 4 based on the eye opening distance acquired by the acquisition unit 11. The calculation unit 12 calculates the degree of eye opening of a passenger aboard the vehicle 3, in this case the driver 4. The degree of eye opening is an index showing the open / closed state of the eyes 5. An eye opening degree of 100% is an awakening state, with the eyes 5 being completely open. An eye opening degree of 25% is a state of smiling, squinting, etc. An eye opening degree of 20% or less is a closed eye state. An eye opening degree of 0% is a state in which the eyes 5 are completely closed.

[0016] Further, the calculation unit 12 sets the eye opening distance at which the eye opening degree of the driver 4 is 100%, i.e., the eye 5 is fully open, based on the eye opening distance of the driver 4. The eye opening degree represents the ratio of the eye opening distance to the eye opening distance at 100% eye opening. The eye opening distance at 100% eye opening is a reference eye opening value. The calculation unit 12 can set the eye opening distance at 100% eye opening by various known methods based on the eye opening distance acquired by the acquisition unit 11. The calculation unit 12 can set the eye opening distance at 100% eye opening based on, for example, the most frequent eye opening distance in the frequency distribution of the eye opening distance within a predetermined time, the average value of the maximum eye opening distance for each predetermined time, or the like.

[0017] The calculation unit 12 may calculate the degree of eye opening of the driver 4 for each of the left eye and the right eye of the driver 4. The degree of eye opening may be an average value of the degree of eye opening of the left eye and the right eye of the driver 4. The calculation unit 12 may calculate the degree of eye opening of only one of the eyes 5 of the driver 4. The calculation unit 12 calculates the degree of eye opening for each of the images continuously acquired from the camera 2.

[0018] The calculation unit 12 also calculates PERCLOS based on the blink information acquired from the driver 4. PERCLOS is a criterion for drowsiness judgment. As shown in FIG. 3, PERCLOS indicates the ratio of eye-closed time per unit time. The eye-closed time t used in calculating PERCLOS is the time during which the degree of eye opening is 20% or less within the unit time. In the example shown in FIG. 3, n blinks are detected within the unit time. Also, in FIG. 3, the eye-closed time during the first blink within the unit time is t1, the eye-closed time during the second blink is t2, the eye-closed time during the n-1 blink is t3, and the eye-closed time during the n-2 blink is t4. n-1 , the eye closure time during the nth blink is t n The eye closure time t used to calculate the park loss is the total value of the eye closure time t within a unit time. The park loss is calculated by dividing the total value of the eye closure time t within a unit time by the unit time. Typically, a relatively large park loss value indicates stronger drowsiness, and a relatively small park loss value indicates weaker drowsiness.

[0019] The determination unit 13 determines the drowsiness of the driver 4 based on the par cross. The determination unit 13 determines whether the driver 4 is in an abnormal state such as drowsiness using the par cross calculated by the calculation unit 12. For example, when the par cross is equal to or greater than a threshold for drowsiness determination, the determination unit 13 determines that the driver 4 is very drowsy, and when the par cross is lower than the threshold for drowsiness determination, the determination unit 13 determines that the driver 4 is very drowsy. This threshold for drowsiness determination is a preset value.

[0020] In addition, the determination unit 13 analyzes the opening and closing movement of the eyes 5 and determines the type of the opening and closing movement. The determination unit 13 can classify the opening and closing movement of the eyes 5 into four types based on the acquired blink information. The four types are composed of shallow eye closure, deep eye closure, shallow blink, and deep blink.

[0021] Shallow eye closure is the opening and closing of the eyes 5 that occurs when the light is too bright or when the eyes are narrowed when laughing. Deep eye closure is the opening and closing of the eyes 5 that occurs during microsleep due to sleepiness or when the eyes are closed. Microsleep refers to a state in which the degree of opening of the eyes 5 is 20% or less for 0.5 seconds or more and less than 3 seconds, when the degree of opening of the eyes 5 is 100% in an awake state. It can be determined that microsleep occurs when the eye closing time t is 0.5 seconds or more and less than 3 seconds. Shallow blinking occurs when the blink is not completely closed due to individual differences, or when information leaks due to the frame rate of the camera 2. Deep blinking is the opening and closing of the eyes 5 that occurs during normal blinking.

[0022] As shown in FIG. 4, when 100% eye opening is considered to be a normal state (awake state), the determination unit 13 determines the state of the occupant based on the duration T of eye closure of the occupant and the minimum degree of eye opening within the duration T of eye closure. The state of the occupant, here the state of the driver 4, includes the opening and closing of the eyes 5. The opening and closing of the eyes 5 is classified into four types: deep blinking, shallow blinking, deep eye closure, and shallow eye closure. In other words, the determination unit 13 determines which of the four types the opening and closing of the eyes 5 corresponds to as the state of the driver 4.

[0023] The duration T of eye closure is the duration of blinking, as shown in FIG. 5. As shown in FIG. 4, a deep blink is an opening / closing action in which the duration T of eye closure is shorter than a predetermined time, and the minimum degree of eye opening during the duration T of eye closure is 20% or less. A shallow blink is an opening / closing action in which the duration T of eye closure is shorter than a predetermined time, and the minimum degree of eye opening during the duration T of eye closure is greater than 20%. A deep eye closure is an opening / closing action in which the duration T of eye closure is greater than a predetermined time, and the minimum degree of eye opening during the duration T of eye closure is greater than 20%. A shallow eye closure is an opening / closing action in which the duration T of eye closure is greater than a predetermined time, and the minimum degree of eye opening during the duration T of eye closure is greater than 20%.

[0024] In addition, the ratio of the opening and closing of the eyes 5 varies from person to person. The opening and closing of the eyes 5 has characteristics for each individual. For example, some people blink deeply and some people blink shallowly. As shown in FIG. 6, in a person who blinks deeply, the ratio of deep blinks is greater than the ratio of shallow blinks in both a normal state and a sleepy state. As shown in FIG. 7, in a person who blinks shallowly, the ratio of shallow blinks is greater than the ratio of deep blinks in both a normal state and a sleepy state. In addition, in common with a person who blinks deeply and a person who blinks shallowly, the ratio of shallow eye closure is smaller than the ratio of shallow blinks in both a normal state and a sleepy state. Therefore, the occupant monitoring device 1 is configured to exclude all shallow eye closure from the sleepiness determination criterion (PERCLOS). Specifically, the calculation unit 12 excludes, from the eye closure time used for calculating the PERCLOS, the time during which the eye closure duration T is equal to or longer than a predetermined time and the minimum eye opening degree is greater than a predetermined value. The eye-closing time is the time during the eye-closing duration T during which the degree of eye opening is equal to or less than a predetermined value.

[0025] The process of setting the drowsiness determination criterion by the passenger monitoring device 1 will be described with reference to Fig. 8. The control shown in Fig. 8 is executed by the passenger monitoring device 1.

[0026] In step S1, blink information of the driver 4 is acquired. The passenger monitoring device 1 acquires the blink information of the driver 4 as information for setting the par cross, which is a drowsiness judgment criterion. The blink information of the driver 4 includes the eye opening distance within a unit time, the number of blinks within a unit time, and the duration of eye closure within a unit time. As illustrated in FIG. 3, this unit time is the unit time that is the denominator in the par cross equation. The acquisition unit 11 calculates the eye opening distance within the unit time according to the unit time used for calculating the par cross. The determination unit 13 judges the presence or absence of blinking within the unit time based on the eye opening distance within the unit time calculated by the acquisition unit 11. For example, the determination unit 13 judges the presence or absence of blinking by various known methods based on the amount of change in the eye opening distance within the unit time and the degree of eye opening calculated by the calculation unit 12. Furthermore, the determination unit 13 detects the number of blinks within the unit time based on the judgment result of the presence or absence of blinking. The calculation unit 12 calculates the duration T of eye closure within a unit time based on the blink detection result by the determination unit 13. When step S1 is executed, the process proceeds to step S2. Note that, in step S1, blink information of the driver 4 needs to be acquired, and there is no particular limitation as to whether the calculation process is performed in step S1 or before step S1. In other words, in step S1, the calculation result of the blink information for the unit time may be acquired.

[0027] In step S2, the blink rate is calculated. The calculation unit 12 calculates the blink rate using the blink information acquired in step S1. Here, the blink rate is calculated using an index that expresses the blink rate in units of time. For example, the blink rate is calculated by dividing the duration of eye closure within a unit time by the number of blinks within the unit time. More specifically, in step S2, a subroutine shown in FIG. 9 is executed. This subroutine will be described later with reference to FIG. 9. After step S2 is executed, the process proceeds to step S3.

[0028] In step S3, the eye closing time during a blink is calculated. The calculation unit 12 calculates the eye closing time during a blink for which the type of opening / closing movement is to be determined. After step S3 is executed, the process proceeds to step S4.

[0029] In step S4, it is determined whether the duration T of eye closure is greater than the blink rate. The determination unit 13 compares the blink rate calculated in step S2 with the duration T of eye closure calculated in step S3, and determines whether the duration T of eye closure is greater than the blink rate. If the determination in step S4 is affirmative, the process proceeds to step S5. If the determination in step S4 is negative, the process proceeds to step S14.

[0030] In step S5, the open / close state of the eyes 5 is determined to be a closed state. The determination unit 13 determines that the type of the opening / closing motion is closed. After step S5 is executed, the process proceeds to step S6.

[0031] In step S6, the minimum eye opening distance during a blink is calculated. The calculation unit 12 calculates the minimum value of the eye opening distance during a blink. After step S6 is executed, the process proceeds to step S7.

[0032] In step S7, the eye opening distance of 100% eye opening degree is acquired. The determination unit 13 acquires the eye opening distance of 100% eye opening degree calculated by the calculation unit 12. After step S7 is executed, the process proceeds to step S8.

[0033] In step S8, the minimum eye opening degree during a blink is calculated. The calculation unit 12 calculates the minimum value of the eye opening degree during a blink. After step S8 is executed, the process proceeds to step S9.

[0034] In step S9, it is determined whether the eye opening degree is 20% or less. The determination unit 13 determines whether the minimum eye opening degree calculated in step S8 is 20% or less. If the determination in step S9 is negative, the process proceeds to step S10. If the determination in step S9 is positive, the process proceeds to step S11.

[0035] In step S10, it is determined that the eye closure is shallow among the four types of open / closed states. The determination unit 13 determines that the eye closure is shallow among the four types of open / close movements. The determination unit 13 also excludes the duration T of the closed eyes determined to be shallow from the value used to calculate the park cross. When step S10 is executed, this control routine ends.

[0036] In step S11, it is determined that the eye is deeply closed among the four types of open / closed states. The determination unit 13 determines that the eye is deeply closed among the four types of open / closed movements. After step S11 is executed, the process proceeds to step S12.

[0037] In step S12, the eye-closure duration T is included in the value used to calculate the PERCLOS. The calculation unit 12 calculates the PERCLOS using this eye-closure duration T. After step S12 is executed, the process proceeds to step S13.

[0038] In step S13, drowsiness determination is performed using the par cross. The determination unit 13 determines the drowsiness of the driver 4 in the above-mentioned manner using the par cross calculated in accordance with the processing of step S12. When step S13 is executed, this control routine ends.

[0039] In step S14, the open / close state of the eyes 5 is determined to be a blinking state. The determination unit 13 determines that the type of the opening / closing motion is a blinking. After step S14 is executed, the process proceeds to step S15.

[0040] In step S15, the minimum eye opening distance during a blink is calculated. The calculation unit 12 calculates the minimum value of the eye opening distance during a blink. After step S15 is executed, the process proceeds to step S16.

[0041] In step S16, the eye opening distance of 100% eye opening degree is acquired. The determination unit 13 acquires the eye opening distance of 100% eye opening degree calculated by the calculation unit 12. After step S16 is executed, the process proceeds to step S17.

[0042] In step S17, the minimum eye opening degree during a blink is calculated. The calculation unit 12 calculates the minimum value of the eye opening degree during a blink. After step S17 is executed, the process proceeds to step S18.

[0043] In step S18, it is determined whether the eye opening degree is 20% or less. The determination unit 13 determines whether the minimum eye opening degree calculated in step S17 is 20% or less. If the determination in step S18 is affirmative, the process proceeds to step S19. If the determination in step S18 is negative, the process proceeds to step S20.

[0044] In step S19, of the four types of open / close states, it is determined that the blink is a deep blink. The determination unit 13 determines that of the four types of open / close movements, it is a deep blink. After step S19 is executed, the process proceeds to step S12.

[0045] In step S20, of the four types of open / close states, it is determined that the blink is shallow. Of the four types of open / close movements, the determination unit 13 determines that the blink is shallow. After step S20 is executed, the process proceeds to step S12.

[0046] The blink rate calculation process will be described with reference to Fig. 9. As shown in Fig. 9, in the process of calculating the blink rate, the duration T of all closed eyes within a unit time is calculated in step S31. The calculation unit 12 calculates all closed eye durations T within a unit time based on the blink information acquired in step S1 shown in Fig. 8. After step S31 is executed, the process proceeds to step S32.

[0047] In step S32, the number of blinks within a unit time is calculated. The calculation unit 12 calculates the number of blinks within a unit time based on the blink information acquired in step S1 shown in Fig. 8. After step S32 is executed, the process proceeds to step S33.

[0048] In step S33, it is determined whether the blink count is 0. The determination unit 13 determines whether the blink count calculated in step S32 is 0. If the determination in step S33 is affirmative, the process proceeds to step S34. If the determination in step S33 is negative, the process proceeds to step S35.

[0049] In step S34, the blink rate is set to 0.5 seconds. The calculation unit 12 sets the blink rate to 0.5 seconds. After step S34 is executed, the process proceeds to step S3 shown in FIG.

[0050] In step S35, the blink rate is set to a value obtained by dividing the duration T of eye closure by the number of blinks. The calculation unit 12 calculates the value obtained by dividing the duration T of eye closure by the number of blinks, and sets the calculated value to the blink rate. After step S35 is executed, the process proceeds to step S3 shown in Fig. 8. The blink rate calculated by the calculation process shown in Fig. 9 is an index expressed in units of time, and corresponds to a value expressed in units of time (predetermined time) used for comparison with the duration T of eye closure to determine whether the eyes are closed or blinking in the process of step S4 shown in Fig. 8.

[0051] As described above, according to the passenger monitoring device 1 of the embodiment, the opening and closing movement of the eyes 5 is classified into four types, and when it is determined that the eyes are closed lightly, the value related to the light eye closure is excluded from the value required for calculating the park crossing. This makes it possible to calculate the park crossing based on an appropriate value. As a result, the passenger monitoring device 1 improves the accuracy of detecting drowsiness when determining the drowsiness of the passenger based on the park crossing.

[0052] Although the configuration for classifying the open / closed state of the eyes 5 of the driver 4 based on the eye opening distance of the driver 4 has been described, the target is not limited to the driver 4, but can be any passenger in the vehicle 3. For example, when two cameras 2 are installed in the driver's seat and the passenger seat, it is possible to capture an image of the face of the passenger in the passenger seat and analyze the opening and closing behavior of the passenger in the passenger seat by the above-mentioned process for setting the drowsiness determination criteria.

[0053] In addition, the predetermined time (blink rate) used for comparison with the duration T of eye closure to determine whether the eyes are closed or blinking is not limited to the example described above, and may be a fixed value obtained, for example, through an actual evaluation experiment that distinguishes between eye closure and blinking. [Explanation of symbols]

[0054] 1: Occupant monitoring device, 2: Camera, 3: Vehicle, 4: Driver, 5: Eye 11: Acquisition section, 12: Calculation section, 13: Judgment section

Claims

1. A calculation unit that calculates an eye opening degree of an occupant who is riding in a vehicle; a determination unit that determines a state of the occupant based on a duration of the occupant's eyes being closed and a minimum degree of eye opening within the duration of the occupant's eyes being closed; An occupant monitoring device comprising:

2. The determination unit determines the drowsiness of the occupant based on a park cross representing a ratio of a closed eye time per unit time, The calculation unit excludes, from the eye closing time used in calculating the park cross, a time during which the eye closing duration is equal to or longer than a predetermined time and the minimum eye opening degree is greater than a predetermined value, The eye-closure time is the time during which the eye opening degree is equal to or less than the predetermined value during the duration of the eye closure.

2. An occupant monitoring device according to claim 1.

Images