Occupant state determination device, occupant state determination system, occupant state determination method, and occupant state determination program

Abstract

A passenger state determination device (1) according to the present disclosure comprises: a face image acquisition unit (11A) that acquires a face image of a passenger of a vehicle; an eye openness calculation unit (12A) that uses the face image as acquired from the face image acquisition unit (11A) to calculate an eye openness that indicates the degree to which the eyes of the passenger are open; a first eye openness determination unit (13A) that uses the eye openness as acquired from the eye openness calculation unit (12A) to determine whether the passenger is in an eyes open state; a second eye openness determination unit (14A) that, upon acquiring determination results that indicate that the passenger is not in the eyes open state from the first eye openness determination unit (13A), uses the eye openness as acquired from the eye openness calculation unit (12A) to determine whether the passenger is in a looking down state or an eyes closed state; and a passenger state determination unit (15A) that uses a looking down state determination ratio, which indicates the percentage of a first period that the second eye openness determination unit (14A) determined the passenger to be in the looking down state, to determine whether the state of the passenger corresponds to a first passenger state that is an inattentive state that is a result of the looking down state.

Description

Technical Field

[0001] The present disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and an occupant state determination program.

Background Art

[0002] There has been developed a technique for imaging the interior of a vehicle, determining an open / closed eye state and a downward viewing state indicating a state in which the line of sight is directed downward, using the shape of the eyes of an occupant extracted from the occupant in the captured image, and detecting that the occupant is operating a mobile phone, a smartphone, or the like.

[0003] For example, in the determination device of Patent Document 1, using the detected position of the driver's eyes, the eye opening degree, which is the distance between the upper and lower eyelids of the driver, is obtained, and a shape parameter associated with a gradual change in the shape of the driver's eyes accompanying changes in the facial expression and movement of the driver is specified. Based on the eye opening degree and the shape parameter, it is determined whether the state is a downward viewing state and whether the eyes are open.

[0004] Specifically, the determination device determines whether the eye opening degree is less than or equal to a closed threshold value. The fact that the eye opening degree is less than or equal to the closed threshold value means that the eyes may be closed. Then, when the eye opening degree is less than or equal to the closed threshold value, the determination device determines whether the specified shape parameter is less than or equal to a predetermined downward viewing threshold value. Then, when the shape parameter is less than or equal to the downward viewing threshold value, the determination device determines that the driver is performing a downward viewing. In such a case, the determination device outputs an alarm for causing the driver to face forward, assuming that although the driver is in an open-eye state, the driver is looking downward at the time of driving, such as operating a smartphone, without looking ahead. Further, when the shape parameter is greater than the downward viewing threshold value, the determination device determines that the driver is in a closed-eye state. In such a case, the determination device outputs an alarm, assuming that there is a high possibility that the driver is dozing off while driving. Here, the closed threshold value is the maximum distance between the upper and lower eyelids indicating a state in which the eyes are closed. The downward viewing threshold value is the maximum value of the shape parameter of the shape of the eyes when the driver is performing a downward viewing.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Here, the occupant looks downward due to various factors. However, the determination device of Patent Document 1 only describes that it uses the eye - opening degree and shape parameters to distinguish between looking downward due to operations such as those on a smartphone and closing eyes due to dozing while driving, and outputs an alarm assigned for each determination result, but does not describe a specific method for distinguishing the downward looks caused by various factors. Therefore, if the determination device of Patent Document 1 determines that the occupant is looking downward, it may output an alarm to the occupant assuming that the occupant is inattentive regardless of the cause of the downward look. That is, when the occupant looks downward, it is determined that the occupant is inattentive regardless of the cause of the downward look, and there is a problem that the detection accuracy of the occupant state decreases.

[0007] The present disclosure has been made to solve such problems, and an object thereof is to provide an occupant state determination device that improves the detection accuracy of the occupant state.

Means for Solving the Problems

[0008] The occupant state determination device according to the present disclosure includes a face image acquisition unit that acquires a face image which is an image of the face of an occupant of a vehicle, an eye opening degree calculation unit that calculates an eye opening degree indicating the degree of opening of the occupant's eyes using the face image acquired by the face image acquisition unit, a first eye opening degree determination unit that determines whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit, and when a determination result indicating that the occupant is not in an open-eye state is acquired from the first eye opening degree determination unit, a second eye opening degree determination unit that determines whether the occupant is in a downward gaze state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit, and an occupant state determination unit that determines whether the state of the occupant corresponds to a first occupant state which is an inattentive state caused by the downward gaze state using a downward gaze state determination rate indicating the ratio of the number of times the second eye opening degree determination unit determines that the occupant is in a downward gaze state during a first period.

Effect of the Invention

[0009] According to the occupant state determination device according to the embodiment of the present disclosure, the detection accuracy of the occupant state can be improved.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Best Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. FIG. 1 is a diagram showing a configuration example of an occupant state determination device 1A according to Embodiment 1. In the present embodiment, it is assumed that the occupant state determination device 1A is mounted on a vehicle. However, the occupant state determination device 1A is not limited to being permanently installed in the vehicle, and may be configured in a portable device that can be brought into the vehicle, such as a portable navigation device. Further, part or all of the occupant state determination device 1A may be configured on a server capable of communicating with the vehicle.

[0012] As shown in FIG. 1, the occupant state determination system 100A includes an occupant state determination device 1A, a camera 2A, and a warning device 3A.

[0013] The camera 2A photographs the interior of the vehicle and is installed at a position where it can photograph the face of the occupant in the driver's seat of the vehicle. The photographing range of the camera 2A may be wide enough to photograph only the driver's seat, or may be wide enough to photograph the faces of the occupants in all seats of the vehicle.

[0014] The occupant state determination device 1A determines the occupant state based on an image of the occupant's face (hereinafter referred to as "face image") photographed by the camera 2A, and outputs the determination result to the warning device 3A. In the present embodiment, the occupant state determination device 1A determines the state of the occupant in the driver's seat photographed by the camera 2A.

[0015] The warning device 3A issues a warning to the occupant according to the determination result of the occupant state determination unit 15A, and includes a speaker that emits a warning sound or a warning message, a display that displays an alarm screen, and the like.

[0016] As shown in FIG. 1, the occupant state determination device 1A includes a determination unit 10A, a face image acquisition unit 11A, and an eye opening degree calculation unit 12A. The determination unit 10A includes a first eye opening degree determination unit 13A, a second eye opening degree determination unit 14A, an occupant state determination unit 15A, and a warning type determination unit 16A.

[0017] The face image acquisition unit 11A acquires the face image of the occupant captured by the camera 2A. The face image acquisition unit 11A extracts the face image of the occupant from the image inside the vehicle captured by the camera 2A using face recognition technology. The face image acquisition unit 11A outputs the face image of the occupant to the eye opening degree calculation unit 12A.

[0018] The eye opening degree calculation unit 12A uses the face image of the occupant acquired from the face image acquisition unit 11A to detect the position of the highest point of the upper eyelid and the position of the lowest point of the lower eyelid. The eye opening degree calculation unit 12A calculates the distance between the position of the highest point of the upper eyelid and the position of the lowest point of the lower eyelid as the eye opening degree indicating the degree of opening of the occupant's eyes. The eye opening degree calculation unit 12A outputs the calculated eye opening degree to the first eye opening degree determination unit 13A and the second eye opening degree determination unit 14A.

[0019] Although an example in which the eye opening degree calculation unit 12A uses the distance between the position of the highest point of the upper eyelid and the position of the lowest point of the lower eyelid as the eye opening degree has been shown, the distance between the position of the highest point of the upper eyelid and the position of the intersection of the vertical line dropped from the highest point of the upper eyelid and the contour line of the lower eyelid may be used as the eye opening degree. Also, the distance between the position of the highest point of the upper eyelid and the position of the intersection of the vertical line dropped from the highest point of the upper eyelid toward the straight line connecting the inner corner and the outer corner of the eye and the contour line of the lower eyelid may be used as the eye opening degree.

[0020] The first eye opening degree determination unit 13A determines whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A. The first eye opening degree determination unit 13A determines whether the eye opening degree is greater than or equal to a first threshold value. Specifically, the first eye opening degree determination unit 13A determines that the occupant is in an open-eye state when the eye opening degree is greater than or equal to the first threshold value, and determines that the occupant is not in an open-eye state when the eye opening degree is less than the first threshold value.

[0021] When the first eye-opening degree determination unit 13A determines that the eye-opening degree is equal to or greater than the first threshold value, it outputs a determination result including information indicating that the occupant is in an eye-open state (hereinafter, the information indicating that the occupant is in an eye-open state is referred to as eye-open state information) to the occupant state determination unit 15A. When the first eye-opening degree determination unit 13A determines that the eye-opening degree is less than the first threshold value, it outputs a determination result including information indicating that the occupant is not in an eye-open state to the second eye-opening degree determination unit 14A.

[0022] When the second eye-opening degree determination unit 14A obtains a determination result including information indicating that the occupant is not in an eye-open state from the first eye-opening degree determination unit 13A, it determines whether the occupant is in a downward-looking state or a closed-eye state by using the eye-opening degree obtained from the eye-opening degree calculation unit 12A. Specifically, the second eye-opening degree determination unit 14A determines whether the eye-opening degree is equal to or greater than the second threshold value. When the second eye-opening degree determination unit 14A determines that the eye-opening degree is equal to or greater than the second threshold value, it determines that the occupant is in a downward-looking state, and when the eye-opening degree is less than the second threshold value, it determines that the occupant is in a closed-eye state. Here, the first threshold value is greater than the second threshold value.

[0023] The second eye-opening degree determination unit 14A outputs a determination result including information indicating that the occupant is in a downward-looking state (hereinafter, the information indicating that the occupant is in a downward-looking state is referred to as downward-looking state information) or information indicating that the occupant is in a closed-eye state (hereinafter, the information indicating that the occupant is in a closed-eye state is referred to as closed-eye state information) to the occupant state determination unit 15A.

[0024] The occupant state determination unit 15A uses the downward-looking state determination rate indicating the ratio of the cases where the second eye-opening degree determination unit 14A determines that the occupant is in a downward-looking state during the first period to determine whether the occupant's state corresponds to the first occupant state, which is an inattentive state caused by the downward-looking state. Here, the first period is a predetermined period. For example, the first period is a predetermined period (e.g., 20 seconds) when the vehicle speed is equal to or higher than a predetermined speed (e.g., 25 [km / h]) and the steering angle is within a predetermined range (e.g., ±20 degrees). Note that this is only an example, and the first period may be appropriately set by the administrator of the occupant state determination device 1A or the like, or may be stored in advance in the occupant state determination device 1A.

[0025] The details of the occupant state determination unit 15A will be described. First, the occupant state determination unit 15A acquires a determination result including eye-opening state information from the first eye-opening determination unit 13A in the first period, a determination result including downward viewing state information from the second eye-opening determination unit 14A in the first period, or a determination result including eye-closed state information from the second eye-opening determination unit 14A in the first period.

[0026] Then, the occupant state determination unit 15A calculates the ratio occupied by the determination result including downward viewing state information acquired in the first period as the downward viewing state determination rate with respect to the determination result including eye-opening state information acquired in the first period, the determination result including downward viewing state information acquired in the first period, and the determination result including eye-closed state information acquired in the first period.

[0027] Then, the occupant state determination unit 15A determines whether the state of the occupant corresponds to a first occupant state, which is an inattentive state caused by the downward viewing state, using the calculated downward viewing state determination rate. The first occupant state is, for example, a state of performing operations such as on a smartphone or the like.

[0028] Specifically, the occupant state determination unit 15A determines whether the downward viewing state determination rate is equal to or higher than a third threshold value. When the downward viewing state determination rate is equal to or higher than the third threshold value, the occupant state determination unit 15A determines that the state of the occupant is the first state. When the downward viewing state determination rate is less than the third threshold value, the occupant state determination unit 15A determines that the state of the occupant is an occupant state other than the first occupant state. Furthermore, even when the occupant state determination unit 15A obtains a determination result including the eye-opening state information from the first eye-opening degree determination unit 13A, it determines that the occupant state is an occupant state other than the first occupant state. Here, the third threshold value is set to, for example, 50 percent. Although an example where the third threshold value is 50 percent has been shown, it is not limited thereto and is appropriately set by the administrator or the like of the occupant state determination device 1A.

[0029] The occupant state determination unit 15A outputs a determination result indicating whether the occupant state corresponds to the first occupant state to the warning type determination unit 16A.

[0030] Here, when the downward viewing state determination rate is equal to or higher than the third threshold value, the occupant state determination unit 15A determines that the occupant state is the first occupant state. When the downward viewing state determination rate is less than the third threshold value, the reason why the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state will be explained.

[0031] Generally, when an occupant operates a smartphone or the like, the line of sight is continuously or frequently directed downward. That is, since the occupant is in a downward viewing state continuously or frequently during the first period, the downward viewing state determination rate tends to be high. Also, due to the occupant being in a downward viewing state continuously or frequently during a predetermined period, the time for directing the line of sight forward is reduced compared to normal driving, so it can be considered that the occupant is inattentive. For the above reasons, when the downward viewing state determination rate is equal to or higher than the third threshold value, the occupant state determination unit 15A determines that the occupant state is the first occupant state, which is an inattentive state caused by the downward viewing state.

[0032] For the above reasons, when the downward viewing state determination rate is equal to or higher than the third threshold value, the occupant state determination unit 15A determines that the occupant state is the first occupant state. When the downward viewing state determination rate is less than the third threshold value, the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state.

[0033] The warning type determination unit 16A determines the type of warning to be output by the warning device 3A that prompts the occupant to give a warning, using the determination result obtained from the occupant state determination unit 15A. When the occupant state determination unit 15A determines that the state of the occupant is the first occupant state, the warning type determination unit 16A causes the warning device 3A to output the first warning. When the occupant state determination unit 15A determines that the state of the occupant is an occupant state other than the first occupant state, the warning type determination unit 16A does not cause the warning device 3A to output the first warning.

[0034] FIG. 2 is a flowchart showing an operation example of the occupant state determination device 1A according to the present embodiment. The processing executed by the occupant state determination device 1A in the present embodiment will be described with reference to the flowchart shown in FIG. 2. Note that the operation of the occupant state determination device 1A is started, for example, after the ignition of the vehicle is turned on. Further, it is preferable to perform the determination process of the occupant state every time a face image is acquired from the camera 2A, that is, for each frame.

[0035] First, the face image acquisition unit 11A acquires a face image (step ST101).

[0036] Next to step ST101, the eye opening degree calculation unit 12A calculates the eye opening degree using the face image acquired from the face image acquisition unit 11A (step ST102).

[0037] Next to step ST102, the first eye opening degree determination unit 13A determines whether the occupant is in an eye - open state using the eye opening degree acquired from the eye opening degree calculation unit 12A (step ST103).

[0038] When the occupant is not in an eye - open state (when step ST103 is NO), the second eye opening degree determination unit 14A determines whether the occupant is in a downward - looking state or a closed - eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A (step ST104).

[0039] Next to step ST104, the occupant state determination unit 15A calculates a downward viewing state determination rate using the determination result including the eye-opening state information acquired from the first eye-opening degree determination unit 13A, the determination result including the downward viewing state information acquired from the second eye-opening degree determination unit 14A, and the determination result including the eye-closed state information acquired from the second eye-opening degree determination unit 14A (step ST105).

[0040] Next to step ST105, the occupant state determination unit 15A determines whether the downward viewing state determination rate is equal to or greater than a third threshold value (step ST106).

[0041] When the downward viewing state determination rate is equal to or greater than the third threshold value (when step ST106 is YES), the occupant state determination unit 15A determines that the state of the occupant is the first occupant state (step ST107).

[0042] When the occupant is in an eye-opening state (when step ST103 is YES), or when the downward viewing state determination rate is less than the third threshold value (when step ST106 is NO), the occupant state determination unit 15A determines that the state of the occupant is an occupant state other than the first occupant state (step ST108).

[0043] Next to step ST107, the warning type determination unit 16A causes the warning device 3A to output a first warning (step ST109).

[0044] Next to step ST108, the warning type determination unit 16A does not cause the warning device 3A to output a first warning (step ST110). Then, the occupant state determination device 1A ends the operation shown in the flowchart of FIG. 2.

[0045] FIG. 3 and FIG. 4 are diagrams showing a hardware configuration example of the occupant state determination device 1A according to Embodiment 1. The functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A in the occupant state determination device 1A are realized by a processing circuit. That is, the occupant state determination device 1A includes a processing circuit for realizing the above functions. The processing circuit may be a processing circuit 100 as dedicated hardware, or may be a processor 101 that executes a program stored in the memory 102.

[0046] As shown in FIG. 3, when the processing circuit is dedicated hardware, the processing circuit 100 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A may be realized by a plurality of processing circuits 100, or the functions of each unit may be collectively realized by one processing circuit 100.

[0047] As shown in FIG. 4, when the processing circuit is the processor 101, the functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 102. The processor 101 realizes the functions of each unit by reading and executing the program stored in the memory 102. That is, the occupant state determination device 1A includes a memory 102 for storing a program that, when executed by the processor 101, causes the steps shown in the flowchart of FIG. 2 to be executed as a result. Also, this program can be said to cause a computer to execute the procedures or methods of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A.

[0048] Here, the processor 101 is a CPU (Central Processing Unit), a processing device, an arithmetic device, a microprocessor, or the like. The memory 102 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or a flash memory, or may be a magnetic disk such as a hard disk or a flexible disk, or may be an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc).

[0049] Note that, regarding the functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A, part of them may be realized by dedicated hardware and part of them may be realized by software or firmware. In this way, the processing circuit in the occupant state determination device 1A can realize the above-mentioned functions by hardware, software, firmware, or a combination thereof.

[0050] The occupant state determination device 1A according to the present embodiment includes a face image acquisition unit 11A that acquires a face image which is an image of the face of an occupant of a vehicle, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree of opening of the occupant's eyes using the face image acquired by the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A when a determination result indicating that the occupant is not in an open-eye state is acquired from the first eye opening degree determination unit 13A, and an occupant state determination unit 15A that determines whether the state of the occupant corresponds to a first occupant state which is an inattentive state caused by the downward-looking state using a downward-looking state determination rate indicating the ratio of the number of times the second eye opening degree determination unit 14A determines that the occupant is in a downward-looking state during a first period. The occupant state determination device 1A can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state based on the determination using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of misjudgment in which an occupant determined to be in a downward-looking state or a closed-eye state is determined to be in the first occupant state although the occupant is not in the first occupant state, and can improve the detection accuracy of the occupant state.

[0051] The occupant state determination device 1A according to this embodiment includes a face image acquisition unit 11A that acquires a face image which is an image of the face of a vehicle occupant, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree of opening of the occupant's eyes using the face image acquired from the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A, and when a determination result indicating that the occupant is not in an open-eye state is acquired from the first eye opening degree determination unit 13A, a second eye opening degree determination unit 14A that determines whether the occupant is in a downward gaze state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A, and an occupant state determination unit 15A that determines whether the state of the occupant corresponds to a first occupant state which is an inattentive state caused by the downward gaze state using a downward gaze state determination rate indicating the ratio of the number of times the second eye opening degree determination unit 14A determines that the occupant is in a downward gaze state during a first period. The occupant state determination unit 15A determines whether the downward gaze state determination rate is greater than or equal to a third threshold value. When the downward gaze state determination rate is greater than or equal to the third threshold value, the occupant state determination unit 15A determines that the state of the occupant is the first occupant state. When the downward gaze state determination rate is less than the third threshold value, the occupant state determination unit 15A determines that the state of the occupant is an occupant state other than the first occupant state. The occupant state determination device 1A can determine whether an occupant determined to be in a downward gaze state or a closed-eye state corresponds to the first occupant state by a determination using the downward gaze state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of an erroneous determination in which an occupant determined not to be in the first occupant state is determined to be in the first occupant state, and can improve the detection accuracy of the occupant state.

[0052] The occupant state determination device 1A according to this embodiment includes a face image acquisition unit 11A that acquires a face image which is an image of the face of an occupant in a vehicle, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree of eye opening of the occupant using the face image acquired from the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant is in an eye-open state using the eye opening degree acquired from the eye opening degree calculation unit 12A, and when the determination result indicating that the occupant is not in an eye-open state is acquired from the first eye opening degree determination unit 13A, a second eye opening degree determination unit 14A that determines whether the occupant is in a downward gaze state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A, and an occupant state determination unit 15A that determines whether the state of the occupant corresponds to a first occupant state which is an inattentive state caused by the downward gaze state using a downward gaze state determination rate indicating the ratio of the cases where the second eye opening degree determination unit 14A determines that the occupant is in a downward gaze state in the first period. Further, the occupant state determination device 1A includes a warning type determination unit 16A that determines the type of warning output by a warning device 3A that prompts a warning to the occupant using the determination result acquired from the occupant state determination unit 15A. When the warning type determination unit 16A acquires a determination result indicating that the state of the occupant is the first occupant state from the occupant state determination unit 15A, the warning type determination unit 16A outputs a first warning to the warning device 3A. With such a configuration, the occupant state determination device 1A can avoid determining that an occupant determined to be in a downward gaze state or a closed-eye state is in the first occupant state even though the occupant is in an occupant state other than the first occupant state, and outputting an unnecessary warning to the occupant. Therefore, the occupant state determination device 1A can output a necessary warning to the occupant.

[0053] The occupant state determination system 100A according to this embodiment includes a face image acquisition unit 11A that acquires a face image, which is an image of the face of a vehicle occupant, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree of opening of the occupant's eyes using the face image acquired by the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A when a determination result indicating that the occupant is not in an open-eye state is acquired from the first eye opening degree determination unit 13A. The occupant state determination system 100A further includes an occupant state determination unit 15A that determines whether the state of the occupant corresponds to a first occupant state, which is an inattentive state caused by the downward-looking state, using a downward-looking state determination rate indicating the ratio of the number of times the second eye opening degree determination unit 14A determines that the occupant is in a downward-looking state during a first period. The occupant state determination system 100A also includes an occupant state determination device 1A, a camera 2A installed in the vehicle so as to be able to image at least the face of the occupant, and a warning device 3A that outputs a warning to the occupant. The occupant state determination device 1A determines the state of the occupant using the face image captured by the camera 2A and causes the warning device 3A to output a warning according to the determination result of the occupant state determination unit 15A. The occupant state determination system 100A can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by a determination using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of false determination in which an occupant determined not to be in the first occupant state is determined to be in the first occupant state, and can improve the detection accuracy of the occupant state.

[0054] The occupant state determination method according to this embodiment includes: a step of acquiring a face image which is an image of the face of an occupant in a vehicle; a step of an eye opening degree calculation unit 12A calculating an eye opening degree indicating the degree of opening of the eyes of the occupant using the face image acquired from a face image acquisition unit 11A; a step of a first eye opening degree determination unit 13A determining whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A; a step of a second eye opening degree determination unit 14A determining whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A when the second eye opening degree determination unit 14A acquires a determination result indicating that the occupant is not in an open-eye state from the first eye opening degree determination unit 13A; and a step of an occupant state determination unit 15A determining whether the state of the occupant corresponds to a first occupant state which is an inattentive state caused by the downward-looking state using a downward-looking state determination rate indicating the ratio of the number of times the second eye opening degree determination unit 14A determines that the occupant is in a downward-looking state during a first period. The occupant state determination method can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by a determination using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of an erroneous determination in which an occupant determined to be in a downward-looking state or a closed-eye state is determined to be in the first occupant state despite not being in the first occupant state, and can improve the detection accuracy of the occupant state.

[0055] The passenger state determination program according to this embodiment causes a computer to execute the following steps: a step in which a face image acquisition unit 11A acquires a face image that is an image of a face of a passenger in a vehicle; a step in which an eye opening degree calculation unit 12A calculates an eye opening degree indicating the degree of opening of the eyes of the passenger using the face image acquired from the face image acquisition unit 11A; a step in which a first eye opening degree determination unit 13A determines whether or not the passenger is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A; a step in which a second eye opening degree determination unit 14A determines whether the passenger is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A when the second eye opening degree determination unit 14A obtains a determination result indicating that the passenger is not in an open-eye state from the first eye opening degree determination unit 13A; and a step in which a passenger state determination unit 15A determines whether the state of the passenger corresponds to a first passenger state, which is an inattentive state caused by the downward-looking state, using a downward-looking state determination rate indicating the ratio of the number of times the second eye opening degree determination unit 14A determines that the passenger is in a downward-looking state during a first period. The passenger state determination program can determine whether a passenger determined to be in a downward-looking state or a closed-eye state corresponds to the first passenger state based on the determination using the downward-looking state determination rate in the passenger state determination unit 15A. Therefore, the passenger state determination device 1A can suppress the occurrence of misjudgments in which a passenger determined to be in a downward-looking state or a closed-eye state is determined to be in the first passenger state even though the passenger is not in the first passenger state, and can improve the detection accuracy of the passenger state.

[0056] Embodiment 2. The passenger state determination device 1B in this embodiment will be described. The passenger state determination device 1A in Embodiment 1 determines the state of the passenger based on the determination result of the eye opening degree and the downward-looking state determination rate calculated from the determination result of the eye opening degree. However, the passenger state determination device 1B in this embodiment is different from Embodiment 1 in that it determines the state of the passenger based on the downward-looking state determination rate calculated from the determination result of the eye opening degree and the closed-eye state determination rate calculated from the determination result of the eye opening degree in addition to the determination result of the eye opening degree.

[0057] FIG. 5 is a diagram showing a configuration example of the passenger state determination device 1B according to this embodiment. As shown in FIG. 5, the occupant state determination system 100B includes a camera 2B, an occupant state determination device 1B, and the camera 2B. Further, the occupant state determination device 1B includes a determination unit 10B, a face image acquisition unit 11B, and an eye opening degree calculation unit 12B. The determination unit 10B includes a first eye opening degree determination unit 13B, a second eye opening degree determination unit 14B, an occupant state determination unit 15B, and a warning type determination unit 16B.

[0058] In the present embodiment, compared with the occupant state determination device 1A of the first embodiment, the operations of the occupant state determination unit 15B and the warning type determination unit 16B are different. Other configurations and operations are the same as those of the first embodiment.

[0059] The occupant state determination unit 15B uses the downward view state determination rate and the closed eye state determination rate indicating the ratio of the second eye opening degree determination unit 14B in the first period determining that the occupant is in a closed eye state, to determine whether the state of the occupant corresponds to any of a first occupant state, a second occupant state which is an inattentive state caused by a closed eye state, and a third occupant state which is a state not in an inattentive state. Here, the second occupant state is, for example, a state of being sleepy, a state of dozing while driving, etc. The third occupant state is, for example, a state of checking a meter, etc., a state of driving normally, etc.

[0060] The details of the occupant state determination unit 15B will be described. First, the occupant state determination unit 15B acquires a determination result including eye opening state information from the first eye opening degree determination unit 13B in the first period, a determination result including downward view state information from the second eye opening degree determination unit 14B in the first period, or a determination result including closed eye state information from the second eye opening degree determination unit 14B in the first period.

[0061] Then, the occupant state determination unit 15B calculates, as the downward view state determination rate, the ratio occupied by the determination result including the downward view state information acquired in the first period, with respect to the determination result including the eye opening state information acquired in the first period, the determination result including the downward view state information acquired in the first period, and the determination result including the closed eye state information acquired in the first period.

[0062] In addition, the passenger state determination unit 15B calculates, as the closed-eye state determination rate, the ratio that the determination result including the closed-eye state information acquired in the first period occupies among the determination result including the eye-opening state information acquired in the first period, the determination result including the downward-looking state information acquired in the first period, and the determination result including the closed-eye state information acquired in the first period.

[0063] Then, the passenger state determination unit 15B uses the downward-looking state determination rate and the closed-eye state determination rate to determine whether the state of the passenger corresponds to any one of the first passenger state, the second passenger state, and the third passenger state.

[0064] Specifically, the passenger state determination unit 15B determines whether the downward-looking state determination rate is equal to or higher than a third threshold value. When the downward-looking state determination rate is equal to or higher than the third threshold value, the passenger state determination unit 15B determines that the state of the passenger is the first state. When the downward-looking state determination rate is less than the third threshold value, the passenger state determination unit 15B determines whether the closed-eye state determination rate is equal to or higher than a fourth threshold value. When the closed-eye state determination rate is equal to or higher than the fourth threshold value, the passenger state determination unit 15B determines that the state of the passenger is the second passenger state. When the closed-eye state determination rate is less than the fourth threshold value, the passenger state determination unit 15B determines that the state of the passenger is the third passenger state. Further, when the passenger state determination unit 15B acquires the determination result including the eye-opening state information from the first eye-opening degree determination unit 13B, the passenger state determination unit 15B also determines that the state of the passenger is the third passenger state. Here, the fourth threshold value is set to, for example, 50 percent. Although an example where the fourth threshold value is 50 percent is shown, it is not limited thereto, and it is appropriately set by the administrator or the like of the passenger state determination device 1B.

[0065] The passenger state determination unit 15B outputs a determination result indicating whether the state of the passenger corresponds to any one of the first passenger state, the second passenger state, and the third passenger state to the warning type determination unit 16B.

[0066] Here, when the downward viewing state determination rate is equal to or higher than the third threshold, the occupant state determination unit 15B determines that the occupant's state is the first occupant state. When the downward viewing state determination rate is less than the third threshold and the eyes-closed state determination rate is equal to or higher than the fourth threshold, the occupant state determination unit 15B determines that the occupant's state is the second occupant state. When the downward viewing state determination rate is less than the third threshold and the eyes-closed state determination rate is less than the fourth threshold, the reason for determining that it is the third occupant state will be described.

[0067] Generally, when an occupant is sleepy or dozing while driving, the occupant closes their eyes continuously or frequently. Therefore, as described above, compared with the first occupant state, the downward viewing state determination rate tends to be lower. Further, since the occupant is in an eyes-closed state continuously or frequently during the first period, the eyes-closed state determination rate tends to be higher. Also, due to the occupant being in an eyes-closed state continuously or frequently during a predetermined period, the time the occupant looks forward is reduced compared to normal driving, so the occupant can be considered to be in an inattentive state. For the above reasons, when the downward viewing state determination rate is less than the third threshold and the eyes-closed state determination rate is equal to or higher than the fourth threshold, the occupant state determination unit 15B determines that the occupant's state is the second occupant state.

[0068] Also, when the occupant checks a meter or the like, the occupant looks downward temporarily or infrequently. Therefore, as described above, compared with the first occupant state, the downward viewing state determination rate tends to be lower. Further, since the occupant is in a downward viewing state temporarily or infrequently during the first period, compared with the second occupant state, the eyes-closed state determination rate tends to be lower. Also, although the occupant is in a downward viewing state temporarily or infrequently during a predetermined period, since the time the occupant looks forward does not decrease compared with the first occupant state or the second occupant state, the occupant can be considered to be in a state where they are not inattentive. For the above reasons, when the downward viewing state determination rate is less than the third threshold and the eyes-closed state determination rate is less than the fourth threshold, the occupant state determination unit 15B determines that the occupant's state is the third occupant state.

[0069] For the above reasons, when the downward view state determination rate is equal to or higher than the third threshold, the occupant state determination unit 15B determines that the occupant's state is the first occupant state. When the downward view state determination rate is less than the third threshold and the eyes-closed state determination rate is equal to or higher than the fourth threshold, the occupant state determination unit 15B determines that the occupant's state is the second occupant state. When the downward view state determination rate is less than the third threshold and the eyes-closed state determination rate is less than the fourth threshold, the occupant state determination unit 15B determines that the occupant's state is the third occupant state.

[0070] The warning type determination unit 16B determines the type of warning to be output by the warning device 3B that prompts the occupant to give a warning, using the determination result obtained from the occupant state determination unit 15B. When the occupant state determination unit 15B determines that the occupant's state is the first state, the warning type determination unit 16B causes the warning device 3B to output the first warning. When the occupant state determination unit 15B determines that the occupant's state is the second state, the warning type determination unit 16B causes the warning device 3B to output a second warning different from the first warning. When the occupant state determination unit 15B determines that the occupant's state is the third state, the warning type determination unit 16B does not cause the warning device 3B to output the first warning and the second warning.

[0071] FIG. 6 is a flowchart showing an operation example of the occupant state determination device 1B according to the present embodiment. The processing executed by the occupant state determination device 1B in the present embodiment will be described with reference to the flowchart shown in FIG. 6. The operation of the occupant state determination device 1B is started, for example, after the ignition of the vehicle is turned on. Further, it is preferable to perform the determination process of the occupant state every time a face image is acquired from the camera 2B, that is, for each frame.

[0072] First, the face image acquisition unit 11B acquires a face image (step ST201).

[0073] Next to step ST201, the eye opening degree calculation unit 12B calculates the eye opening degree using the face image acquired from the face image acquisition unit 11B (step ST202).

[0074] Next to step ST202, the first eye-opening degree determination unit 13B determines whether the occupant is in an eye-open state by using the eye-opening degree obtained from the eye-opening degree calculation unit 12B (step ST203).

[0075] When the occupant is not in an eye-open state (when step ST203 is NO), the second eye-opening degree determination unit 14B determines whether the occupant is in a downward-looking state or an eye-closed state by using the eye-opening degree obtained from the eye-opening degree calculation unit 12B (step ST204).

[0076] Next to step ST204, the occupant state determination unit 15B calculates a downward-looking state determination rate and an eye-closed state determination rate by using the determination result including the eye-open state information obtained from the first eye-opening degree determination unit 13B, the determination result including the downward-looking state information obtained from the second eye-opening degree determination unit 14B, and the determination result including the eye-closed state information obtained from the second eye-opening degree determination unit 14B (step ST205).

[0077] Next to step ST205, the occupant state determination unit 15B determines whether the downward-looking state determination rate is equal to or greater than a third threshold value (step ST206).

[0078] When the downward-looking state determination rate is equal to or greater than the third threshold value (when step ST206 is YES), the occupant state determination unit 15B determines that the state of the occupant is the first occupant state (step ST207).

[0079] When the downward-looking state determination rate is less than the third threshold value (when step ST206 is NO), the occupant state determination unit 15B determines whether the eye-closed state determination rate is equal to or greater than a fourth threshold value (step ST208).

[0080] When the eye-closed state determination rate is equal to or greater than the fourth threshold value (when step ST208 is YES), the occupant state determination unit 15B determines that the state of the occupant is the second occupant state (step ST209).

[0081] When the occupant is in an awake state (when step ST203 is YES), or when the closed-eye state determination rate is less than the fourth threshold (when step ST208 is NO), the occupant state determination unit 15B determines that the state of the occupant is the third occupant state (step ST210).

[0082] Next to step ST207, the warning type determination unit 16B causes the warning device 3B to output the first warning (step ST211).

[0083] Next to step ST209, the warning type determination unit 16B causes the warning device 3B to output the second warning (step ST212).

[0084] Next to step ST210, the warning type determination unit 16B does not cause the warning device 3B to output the first warning and the second warning (step ST213). Then, the occupant state determination device 1B ends the operation shown in the flowchart of FIG. 6.

[0085] The occupant state determination device 1B according to this embodiment includes a face image acquisition unit 11B that acquires a face image which is an image of the face of an occupant of a vehicle, an eye opening degree calculation unit 12B that calculates an eye opening degree indicating the degree of opening of the occupant's eyes using the face image acquired from the face image acquisition unit 11B, a first eye opening degree determination unit 13B that determines whether the occupant is in an eye-open state using the eye opening degree acquired from the eye opening degree calculation unit 12B, and when a determination result indicating that the occupant is not in an eye-open state is acquired from the first eye opening degree determination unit 13B, a second eye opening degree determination unit 14B that determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12B, and an occupant state determination unit 15B that determines whether the state of the occupant corresponds to a first occupant state which is an inattentive state caused by the downward-looking state, using a downward-looking state determination rate indicating the ratio of the second eye opening degree determination unit 14B determining that the occupant is in a downward-looking state during a first period. The occupant state determination unit 15B uses, in addition to the downward-looking state determination rate, a closed-eye state determination rate indicating the ratio of the second eye opening degree determination unit 14B determining that the occupant is in a closed-eye state during the first period, to determine whether the state of the occupant corresponds to any one of a first occupant state, a second occupant state which is an inattentive state caused by the closed-eye state, and a third occupant state which is not an inattentive state. With such a configuration, in addition to the effects described in Embodiment 1, the occupant state determination device 1B can detect more occupant states compared to the occupant state determination device 1A of Embodiment 1 by detecting a second occupant state and a third occupant state in addition to the first occupant state.

[0086] The occupant state determination device 1B according to this embodiment includes a face image acquisition unit 11B that acquires a face image, which is an image of the face of an occupant of a vehicle, an eye opening degree calculation unit 12B that calculates an eye opening degree indicating the degree of opening of the occupant's eyes using the face image acquired by the face image acquisition unit 11B, a first eye opening degree determination unit 13B that determines whether the occupant is in an open-eye state using the eye opening degree acquired by the eye opening degree calculation unit 12B, and when the determination result indicating that the occupant is not in an open-eye state is acquired from the first eye opening degree determination unit 13B, a second eye opening degree determination unit 14B that determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired by the eye opening degree calculation unit 12B, and an occupant state determination unit 15B that determines whether the state of the occupant corresponds to a first occupant state, which is an inattentive state caused by the downward-looking state of the occupant, using a downward-looking state determination rate indicating the ratio of the number of times the second eye opening degree determination unit 14B determines that the occupant is in a downward-looking state during a first period. The occupant state determination device 1B further includes a warning type determination unit 16B that determines the type of warning output by a warning device 3B that prompts a warning to the occupant using the determination result acquired from the occupant state determination unit 15B. When the warning type determination unit 16B acquires a determination result indicating that the state of the occupant is the first occupant state from the occupant state determination unit 15B, the warning type determination unit 16B outputs a first warning to the warning device 3B. When the warning type determination unit 16B acquires a determination result indicating that the state of the occupant is a second occupant state different from the first occupant state from the occupant state determination unit 15B, the warning type determination unit 16B outputs a second warning different from the first warning to the warning device 3B. With such a configuration, in addition to the effects described in the first embodiment, the occupant state determination device 1B can output more warnings corresponding to the occupant state by outputting warnings corresponding to a second occupant state in addition to the first occupant state, as compared with the case of outputting only warnings corresponding to the first occupant state.

[0087] Embodiment 3. The occupant state determination device 1C in this embodiment will be described. The occupant state determination device 1A in Embodiment 1 determines the state of the occupant based on the determination result of the eye opening degree and the downward viewing state determination rate calculated from the determination result of the eye opening degree. However, the occupant state determination device 1C in the present embodiment differs from Embodiment 1 in that it determines the state of the occupant based on the determination result of the eye opening degree and the downward viewing state determination rate calculated from the determination result of the eye opening degree, in addition to the determination result of the difference in the number of blinks.

[0088] FIG. 7 is a diagram showing a configuration example of the occupant state determination device 1C according to the embodiment. As shown in FIG. 7, the occupant state determination system 100C includes a camera 2C, an occupant state determination device 1C, and the camera 2C. The occupant state determination device 1C includes a determination unit 10C, a face image acquisition unit 11C, an eye opening degree calculation unit 12C, a blink frequency calculation unit 17C, and a reference blink frequency estimation unit 18C. The determination unit 10C includes a first eye opening degree determination unit 13C, a second eye opening degree determination unit 14C, a blink frequency determination unit 19C, an occupant state determination unit 15C, and a warning type determination unit 16C.

[0089] In the present embodiment, compared with the occupant state determination device 1A of Embodiment 1, it further includes a blink frequency calculation unit 17C, a reference blink frequency estimation unit 18C, and a blink frequency determination unit 19C. In the following description, the blink frequency calculation unit 17C, the reference blink frequency estimation unit 18C, the blink frequency determination unit 19C, and the occupant state determination unit 15C whose functions are added to the configuration of Embodiment 1 will be described. The other configurations and operations are the same as those of Embodiment 1.

[0090] The blink frequency calculation unit 17C calculates a blink frequency indicating the number of blinks of the occupant in the second period using the eye opening degree acquired from the eye opening degree calculation unit 12C. Here, the second period is a predetermined period. For example, the second period is a predetermined period (for example, 5 seconds) when the vehicle speed is equal to or higher than a predetermined speed (for example, 25 [km / h]) and the steering angle is within a predetermined range (for example, ±20 degrees). Note that this is only an example, and the second period may be appropriately set by the administrator or the like of the occupant state determination device 1C, or may be stored in advance in the occupant state determination device 1C.

[0091] An example of a method for calculating the number of blinks is shown. The blink count calculation unit 17C determines that the state is a closed-eye state when the eye opening degree falls below the second threshold value, and determines that the state is an open-eye state when the eye opening degree exceeds the first threshold value that is greater than the second threshold value. After detecting the closed-eye state, the blink count calculation unit 17C determines that "there is a blink" when an open-eye state is detected within the third period. When an open-eye state is not detected within the third period after detecting the closed-eye state, the blink count calculation unit 17C determines that "there is no blink". In this way, the blink count calculation unit 17C calculates the number of blinks in the second period. Here, the third period is a predetermined period. For example, the third period is a predetermined period (for example, 0.2 seconds) when the vehicle speed is equal to or higher than a predetermined speed (for example, 25 [km / h]) and the steering angle is within a predetermined range (for example, ±20 degrees). Note that this is only an example, and the third period may be appropriately set by the administrator of the occupant state determination device 1C or the like, or may be stored in advance in the occupant state determination device 1C.

[0092] Then, the blink count calculation unit 17C outputs the calculated number of blinks to the reference blink count estimation unit 18C and the blink count determination unit 19C.

[0093] The reference blink count estimation unit 18C estimates, as the reference blink count, the number of blinks that serves as a reference for the occupant using the number of blinks calculated by the blink count calculation unit 17C in the fourth period. Here, the fourth period is a predetermined period. For example, the fourth period is a predetermined period (for example, 10 seconds) when the vehicle speed is equal to or higher than a predetermined speed (for example, 25 [km / h]) and the steering angle is within a predetermined range (for example, ±20 degrees). Note that this is only an example, and the fourth period may be appropriately set by the administrator of the occupant state determination device 1C or the like, or may be stored in advance in the occupant state determination device 1C.

[0094] Then, the reference blink rate estimation unit 18C outputs the estimated reference blink rate to the blink rate determination unit 19C.

[0095] The blink rate determination unit 19C compares the blink rate obtained from the blink rate calculation unit 17C with the reference blink rate obtained from the reference blink rate estimation unit 18C, and determines whether there is a change in the blink rate of the occupant using the blink rate difference indicating the difference between the blink rate and the reference blink rate. Specifically, the blink rate determination unit 19C determines whether the blink rate difference is greater than or equal to a fifth threshold value. When the blink rate difference is greater than or equal to the fifth threshold value, the blink rate determination unit 19C determines that there is a change in the blink rate of the occupant, and when the blink rate difference is less than the fifth threshold value, the blink rate determination unit 19C determines that there is no change in the blink rate of the occupant. Note that the fifth threshold value may be appropriately set by an administrator or the like of the occupant state determination device 1C, or may be stored in advance in the occupant state determination device 1C.

[0096] When the blink rate difference is less than the fifth threshold value, the blink rate determination unit 19C outputs a determination result including information indicating that the blink rate difference is less than the fifth threshold value to the occupant state determination unit 15C. When the blink rate difference is greater than or equal to the fifth threshold value, the blink rate determination unit 19C outputs a determination result including information indicating that the blink rate difference is greater than or equal to the fifth threshold value to the occupant state determination unit 15C.

[0097] Here, the reason why the occupant state determination device 1C of the present embodiment includes the blink rate calculation unit 17C, the reference blink rate estimation unit 18C, and the blink rate determination unit 19C will be described. Generally, when an occupant operates a smartphone or the like, when the occupant is sleepy, when the occupant is dozing while driving, or when the occupant checks a meter or the like, the number of blinks tends to increase or decrease compared to the number of blinks during normal driving. Therefore, when the first eye-opening determination unit 13C determines that the occupant is not in an eye-open state, and when the blink frequency determination unit 19C further determines that there is a change in the blink frequency of the occupant, it is more likely that the occupant is in a state of operating a smartphone or the like, a state of being sleepy, a state of dozing while driving, or a state of checking a meter or the like. In other words, by further determining whether there is a change in the blink frequency of the occupant by the blink frequency determination unit 19C, it is possible to more accurately distinguish between a state in which the occupant is operating a smartphone or the like, a state in which the occupant is sleepy, a state of dozing while driving, or a state in which the occupant is checking a meter or the like, and other occupant states.

[0098] When the occupant state determination unit 15C obtains a determination result indicating that there is a change in the blink frequency of the occupant from the blink frequency determination unit 19C, the occupant state determination unit 15C determines whether the state of the occupant corresponds to the first occupant state by using a downward viewing state determination rate indicating the ratio of the second eye-opening determination unit 14C determining that the occupant is in a downward viewing state during the first period.

[0099] Specifically, the occupant state determination unit 15C determines whether the downward viewing state determination rate is equal to or greater than a third threshold value. When the downward viewing state determination rate is equal to or greater than the third threshold value, the occupant state determination unit 15C determines that the state of the occupant is the first state. When the downward viewing state determination rate is less than the third threshold value, the occupant state determination unit 15C determines that the state of the occupant is an occupant state other than the first occupant state. Here, the occupant state other than the first occupant state includes the second occupant state or the third occupant state. Furthermore, when the occupant state determination unit 15C obtains a determination result including eye-opening state information from the first eye-opening determination unit 13C, or when the occupant state determination unit 15C obtains a determination result indicating that there is no change in the blink frequency of the occupant from the blink frequency determination unit 19C, the occupant state determination unit 15C also determines that the state of the occupant is an occupant state other than the first occupant state.

[0100] The occupant state determination unit 15C outputs a determination result indicating whether the state of the occupant corresponds to the first occupant state to the warning type determination unit 16C.

[0101] FIG. 8 is a flowchart showing an operation example of the occupant state determination device 1C according to the present embodiment. The processing executed by the occupant state determination device 1C in the present embodiment will be described with reference to the flowchart shown in FIG. 8. Note that the operation of the occupant state determination device 1C is started, for example, after the ignition of the vehicle is turned on. Also, it is preferable to perform the determination process of the occupant state every time a face image is acquired from the camera 2C, that is, for each frame.

[0102] First, the face image acquisition unit 11C acquires a face image (step ST301).

[0103] Next to step ST301, the eye opening degree calculation unit 12C calculates the eye opening degree using the face image acquired from the face image acquisition unit 11C (step ST302).

[0104] Next to step ST302, the blink count calculation unit 17C calculates the blink count using the eye opening degree acquired from the eye opening degree calculation unit 12C (step ST303).

[0105] Next to step ST303, the reference blink count estimation unit 18C estimates the blink count as the reference blink count using the blink count calculated by the blink count calculation unit 17C (step ST304).

[0106] Next to step ST304, the first eye opening degree determination unit 13C determines whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12C (step ST305).

[0107] When the occupant is not in the open-eye state (when step ST305 is NO), the second eye opening degree determination unit 14C determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12C (step ST306).

[0108] Next to step ST306, the blink count determination unit 19C determines whether there is a change in the blink count of the occupant (step ST307).

[0109] When there is a change in the number of blinks of the occupant (when step ST307 is YES), the occupant state determination unit 15C calculates a downward viewing state determination rate using the determination result including the eye-opening state information acquired from the first eye-opening degree determination unit 13C, the determination result including the downward viewing state information acquired from the second eye-opening degree determination unit 14C, and the determination result including the eye-closed state information acquired from the second eye-opening degree determination unit 14C (step ST308).

[0110] Next to step ST308, the occupant state determination unit 15C determines whether the downward viewing state determination rate is greater than or equal to a third threshold (step ST309).

[0111] When the downward viewing state determination rate is greater than or equal to the third threshold (when step ST309 is YES), the occupant state determination unit 15C determines that the state of the occupant is the first occupant state (step ST310).

[0112] When the occupant is in an eye-open state (when step ST305 is YES), when there is no change in the number of blinks of the occupant (when step ST307 is NO), or when the downward viewing state determination rate is less than the third threshold (when step ST309 is NO), the occupant state determination unit 15C determines that the state of the occupant is an occupant state other than the first occupant state (step ST311).

[0113] Next to step ST310, the warning type determination unit 16C causes the warning device 3C to output a first warning (step ST312).

[0114] Next to step ST311, the warning type determination unit 16C does not cause the warning device 3C to output a first warning (step ST313). Then, the occupant state determination device 1C ends the operation shown in the flowchart of FIG. 8.

[0115] The occupant state determination device 1C according to a modification of the present embodiment includes a face image acquisition unit 11C that acquires a face image which is an image of the face of an occupant in a vehicle, an eye opening degree calculation unit 12C that calculates an eye opening degree indicating the degree of opening of the occupant's eyes using the face image acquired by the face image acquisition unit 11C, a first eye opening degree determination unit 13C that determines whether the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12C, and when a determination result indicating that the occupant is not in an open-eye state is acquired from the first eye opening degree determination unit 13C, a second eye opening degree determination unit 14C that determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12C, and an occupant state determination unit 15C that determines whether the state of the occupant corresponds to a first occupant state which is an inattentive state caused by the downward-looking state of the occupant using a downward-looking state determination rate indicating the ratio of the number of times the second eye opening degree determination unit 14C determines that the occupant is in a downward-looking state during a first period. Further, the occupant state determination device 1C includes a blink count determination unit 19C that determines whether there is a change in the blink count indicating the number of blinks of the occupant. When the occupant state determination unit 15C acquires a determination result indicating that there is a change in the blink count of the occupant from the blink count determination unit 19C, the occupant state determination unit 15C determines whether the state of the occupant corresponds to the first occupant state using the downward-looking state determination rate. When the occupant state determination unit 15C acquires a determination result indicating that there is no change in the blink count of the occupant from the blink count determination unit 19C, the occupant state determination unit 15C does not determine whether the state of the occupant corresponds to the first occupant state using the downward-looking state determination rate. With such a configuration, the occupant state determination device 1C can more accurately distinguish between a state where the occupant is operating a smartphone or the like, a state where the occupant is sleepy, a state where the occupant is dozing while driving, or a state where the occupant is checking a meter or the like, and other occupant states. As a result, the occupant state determination device 1C can further improve the detection accuracy of the occupant state as compared with the occupant state determination device 1A of the first embodiment.

[0116] Note that, in the present embodiment, the combination of the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C, which are configurations specific to the present embodiment, and the first embodiment has been described. However, the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C, which are configurations specific to the present embodiment, may be combined with the second embodiment.

Explanation of Signs

[0117] 1A Occupant state determination device, 2A Camera, 3A Warning device, 10A Determination unit, 11A Facial image acquisition unit, 12A Eye opening degree calculation unit, 13A First eye opening degree determination unit, 14A Second eye opening degree determination unit, 15A Occupant state determination unit, 16A Warning type determination unit, 100A Occupant state determination system

Claims

1. A face image acquisition unit that acquires a face image that is an image of the face of a vehicle occupant; An eye opening degree calculation unit that calculates an eye opening degree indicating the degree of eye opening of the occupant using the face image acquired from the face image acquisition unit; A first eye opening degree determination unit that determines whether the occupant is in an eye-open state using the eye opening degree acquired from the eye opening degree calculation unit; When a determination result indicating that the occupant is not in an eye-open state is acquired from the first eye opening degree determination unit, a second eye opening degree determination unit that determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit; A passenger state determination unit that determines whether the state of the passenger corresponds to a first passenger state that is a careless state caused by the downward-looking state using a downward-looking state determination rate indicating the ratio of the number of times the second eye opening degree determination unit determines that the passenger is in the downward-looking state during a first period; A passenger state determination device comprising:

2. The eye opening degree is the distance between the highest point of the upper eyelid detected from the face image and the lowest point of the lower eyelid detected from the face image. The passenger state determination device according to claim 1.

3. The first eye opening degree determination unit determines whether the eye opening degree is greater than or equal to a first threshold value. When the eye opening degree is greater than or equal to the first threshold value, the first eye opening degree determination unit determines that the occupant is in the eye-open state, and when the eye opening degree is less than the first threshold value, the first eye opening degree determination unit determines that the occupant is not in the eye-open state. The second eye opening degree determination unit determines whether the eye opening degree is greater than or equal to a second threshold value. When the eye opening degree is greater than or equal to the second threshold value, the second eye opening degree determination unit determines that the occupant is in the downward-looking state, and when the eye opening degree is less than the second threshold value, the second eye opening degree determination unit determines that the occupant is in the closed-eye state. The first threshold value is greater than the second threshold value. The passenger state determination device according to claim 1.

4. The occupant state determination unit determines whether or not the downward view state determination rate is equal to or greater than a third threshold value, When the downward view state determination rate is equal to or greater than the third threshold value, the occupant state determination unit determines that the state of the occupant is the first occupant state, When the downward view state determination rate is less than the third threshold value, the occupant state determination unit determines that the state of the occupant is an occupant state other than the first occupant state. The occupant state determination device according to any one of claims 1 to 3.

5. In addition to the downward view state determination rate, the occupant state determination unit uses a closed-eye state determination rate indicating the ratio of the second eye-opening determination unit determining that the occupant is in the closed-eye state during the first period, and determines whether the state of the occupant corresponds to any one of the first occupant state, a second occupant state which is an inattentive state caused by the closed-eye state, and a third occupant state which is not the inattentive state. The occupant state determination device according to claim 1.

6. The occupant state determination unit determines whether or not the downward view state determination rate is equal to or greater than a third threshold value, When the downward view state determination rate is equal to or greater than the third threshold value, the occupant state determination unit determines that the state of the occupant is the first occupant state, When the downward view state determination rate is less than the third threshold value, the occupant state determination unit determines whether or not the closed-eye state determination rate is equal to or greater than a fourth threshold value, When the closed-eye state determination rate is equal to or greater than the fourth threshold value, the occupant state determination unit determines that the state of the occupant is the second occupant state, When the closed-eye state determination rate is less than the fourth threshold value, the occupant state determination unit determines that the state of the occupant is the third occupant state. The occupant state determination device according to claim 5.

7. A warning type determination unit that determines the type of warning output by a warning device that prompts a warning to the occupant using the determination result obtained from the occupant state determination unit, When the warning type determination unit obtains a determination result indicating that the state of the occupant from the occupant state determination unit is the first occupant state, the warning type determination unit causes the warning device to output a first warning. The occupant state determination device according to claim 1 or 2.

8. A warning type determination unit that determines the type of warning output by a warning device that prompts a warning to the occupant using the determination result obtained from the occupant state determination unit, When the warning type determination unit obtains a determination result indicating that the state of the occupant from the occupant state determination unit is the first occupant state, the warning type determination unit causes the warning device to output a first warning. When the warning type determination unit obtains a determination result indicating that the state of the occupant from the occupant state determination unit is the second occupant state, the warning type determination unit causes the warning device to output a second warning different from the first warning. The occupant state determination device according to claim 5.

9. A blink number determination unit that determines whether there is a change in the number of blinks of the occupant, comprising When the occupant state determination unit obtains a determination result indicating that there is a change in the number of blinks of the occupant from the blink number determination unit, the occupant state determination unit uses the downward viewing state determination rate to determine whether the state of the occupant corresponds to the first occupant state. When the occupant state determination unit obtains a determination result indicating that there is no change in the number of blinks of the occupant from the blink number determination unit, the occupant state determination unit does not use the downward viewing state determination rate to determine whether the state of the occupant corresponds to the first occupant state. The occupant state determination device according to claim 1.

10. A blink number determination unit that determines whether there is a change in the number of blinks of the occupant, comprising When the occupant state determination unit obtains a determination result indicating that there is a change in the blink count of the occupant from the blink count determination unit, it uses the downward gaze state determination rate and the closed-eye state determination rate to determine whether the state of the occupant corresponds to any of the first occupant state, the second occupant state, and the third occupant state. When the occupant state determination unit obtains a determination result indicating that there is no change in the blink count of the occupant from the blink count determination unit, it does not use the downward gaze state determination rate and the closed-eye state determination rate to determine whether the state of the occupant corresponds to any of the first occupant state, the second occupant state, and the third occupant state. The occupant state determination device according to claim 5.

11. A blink count calculation unit that calculates the blink count in a second period using the eye opening degree obtained from the eye opening degree calculation unit, In a fourth period, a reference blink count estimation unit that estimates the reference blink count of the occupant as the reference blink count using the blink count calculated by the blink count calculation unit, and The blink count determination unit compares the blink count obtained from the blink count calculation unit with the reference blink count obtained from the reference blink count estimation unit, and uses a blink count difference indicating the difference between the blink count and the reference blink count to determine whether there is a change in the blink count. The occupant state determination device according to claim 9.

12. The blink count determination unit determines whether the blink count difference is greater than or equal to a fifth threshold value. When the blink count difference is greater than or equal to the fifth threshold value, the blink count determination unit determines that there is a change in the blink count of the occupant, and when the blink count difference is less than the fifth threshold value, the blink count determination unit determines that there is no change in the blink count of the occupant. The occupant state determination device according to claim 11.

13. An abnormal posture determination system comprising an occupant state determination device according to any one of claims 1 to 3, claim 5, claim 6, and claims 8 to 12, a camera installed in the vehicle so as to be able to photograph at least the face of the occupant, and a warning device that outputs a warning to the occupant. The occupant state determination device uses the face image captured by the camera to determine the state of the occupant, and causes the warning device to output a warning according to the determination result of the occupant state determination unit. An occupant state determination system.

14. An occupant state determination method executed by an occupant state determination device, A step in which a face image acquisition unit acquires a face image that is an image of the face of an occupant of a vehicle, and a step in which an eye opening degree calculation unit calculates an eye opening degree indicating the degree of opening of the eyes of the occupant using the face image acquired from the face image acquisition unit. A step in which a first eye opening degree determination unit determines whether or not the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit. When a second eye opening degree determination unit obtains a determination result indicating that the occupant is not in an open-eye state from the first eye opening degree determination unit, the second eye opening degree determination unit uses the eye opening degree acquired from the eye opening degree calculation unit to determine whether the occupant is in a downward gaze state or a closed-eye state. A step in which an occupant state determination unit determines whether the state of the occupant corresponds to a first occupant state, which is an inattentive state caused by the downward gaze state, using a downward gaze state determination rate indicating the ratio of the second eye opening degree determination unit determining that the occupant is in the downward gaze state during a first period. An occupant state determination method comprising the above.

15. A step in which a face image acquisition unit acquires a face image that is an image of the face of an occupant of a vehicle, and a step in which an eye opening degree calculation unit calculates an eye opening degree indicating the degree of opening of the eyes of the occupant using the face image acquired from the face image acquisition unit. A step in which a first eye opening degree determination unit determines whether or not the occupant is in an open-eye state using the eye opening degree acquired from the eye opening degree calculation unit. When the second eye opening degree determination unit obtains a determination result indicating that the occupant is not in an eye-open state from the first eye opening degree determination unit, a step of determining whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree obtained from the eye opening degree calculation unit; A step in which the occupant state determination unit determines whether the state of the occupant corresponds to a first occupant state that is a careless state caused by the downward-looking state, using a downward-looking state determination rate indicating the ratio of the second eye opening degree determination unit determining that the occupant is in the downward-looking state during the first period; An occupant state determination program for causing a computer to execute.

Images