Passenger state determination device, passenger state determination system, passenger state determination method, and passenger state determination program

Abstract

A passenger state determination device (1) according to the present disclosure comprises: a face image acquisition unit (11A) that acquires a face image of a passenger of a vehicle; an eye openness calculation unit (12A) that uses the face image as acquired from the face image acquisition unit (11A) to calculate an eye openness that indicates the degree to which the eyes of the passenger are open; a first eye openness determination unit (13A) that uses the eye openness as acquired from the eye openness calculation unit (12A) to determine whether the passenger is in an eyes open state; a second eye openness determination unit (14A) that, upon acquiring determination results that indicate that the passenger is not in the eyes open state from the first eye openness determination unit (13A), uses the eye openness as acquired from the eye openness calculation unit (12A) to determine whether the passenger is in a looking down state or an eyes closed state; and a passenger state determination unit (15A) that uses a looking down state determination ratio, which indicates the percentage of a first period that the second eye openness determination unit (14A) determined the passenger to be in the looking down state, to determine whether the state of the passenger corresponds to a first passenger state that is an inattentive state that is a result of the looking down state.

Description

Occupant state determination device, occupant state determination system, occupant state determination method, and occupant state determination program

[0001] The present disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and an occupant state determination program.

[0002] A technology has been developed that captures images of the interior of a vehicle and uses the shape of the occupants' eyes extracted from the captured image to determine whether they are open or closed, and whether they are looking downward, thereby detecting whether the occupant is operating a mobile phone, smartphone, etc.

[0003] For example, the determination device in Patent Document 1 uses the detected position of the driver's eyes to determine the degree of eye opening, which is the distance between the driver's upper and lower eyelids, and identifies shape parameters associated with gradual changes in the shape of the driver's eyes that accompany changes in the driver's facial expression and movements.Based on the degree of eye opening and the shape parameters, it determines whether the driver is looking downward and whether their eyes are open.

[0004] Specifically, the determination device determines whether the degree of eye opening is equal to or less than a closure threshold. A degree of eye opening equal to or less than the closure threshold indicates that the eyes are likely closed. If the degree of eye opening is equal to or less than the closure threshold, the determination device determines whether the identified shape parameter is equal to or less than a predetermined downward gaze threshold. If the shape parameter is equal to or less than the downward gaze threshold, the determination device determines that the driver is looking downward. In such a case, the determination device determines that the driver has their eyes open, but is looking downward while driving, for example, by operating a smartphone, rather than looking ahead, and outputs a warning to encourage the driver to look forward. Furthermore, if the shape parameter is greater than the downward gaze threshold, the determination device determines that the driver has their eyes closed. In such a case, the determination device outputs a warning, indicating that the driver is likely drowsy at the wheel. Here, the closure threshold is the maximum distance between the upper and lower eyelids indicating that the eyes are closed. Furthermore, the downward gaze threshold is the maximum value of the shape parameter of the eye shape when the driver is looking downward.

[0005] Japanese Patent Application Laid-Open No. 2016-115117

[0006] Here, occupants may look downward due to various factors. However, the determination device of Patent Document 1 merely describes using the degree of eye opening and shape parameters to distinguish between downward gaze due to operation of a smartphone or the like and closed eyes due to drowsy driving, and outputs an alert assigned to each determination result. However, it does not describe a specific method for distinguishing between downward gaze caused by various factors. Therefore, if the determination device of Patent Document 1 determines that an occupant is looking downward, it may determine that the occupant is inattentive regardless of the cause of the downward gaze and output an alert to the occupant. In other words, when an occupant looks downward, it is determined that the occupant is inattentive regardless of the cause of the downward gaze, which poses a problem of reduced accuracy in detecting the occupant's state.

[0007] The present disclosure has been made to solve such problems, and has an object to provide an occupant state determination device that improves the accuracy of detecting the occupant state.

[0008] The occupant state determination device according to the present disclosure includes a facial image acquisition unit that acquires a facial image, which is an image of the face of a vehicle occupant; an eye opening degree calculation unit that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the facial image acquired from the facial image acquisition unit; a first eye opening degree determination unit that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit; a second eye opening degree determination unit that, when a determination result indicating that the occupant is not in an eye opening state is obtained from the first eye opening degree determination unit, determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit; and an occupant state determination unit that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward-looking state, using a downward-looking state determination rate that indicates the proportion of times the second eye opening degree determination unit has determined that the occupant is in a downward-looking state during a first period.

[0009] According to the occupant state determination device according to the embodiment of the present disclosure, it is possible to improve the accuracy of detecting the occupant state.

[0010] FIG. 1 is a diagram showing an example of the configuration of an occupant state determination device according to embodiment 1. FIG. 2 is a flowchart showing an example of the operation of the occupant state determination device according to embodiment 1. FIG. 3 is a diagram showing a hardware configuration of an occupant state determination device according to embodiment 1. FIG. 4 is a diagram showing a hardware configuration of an occupant state determination device according to embodiment 1. FIG. 5 is a diagram showing an example of the configuration of an occupant state determination device according to embodiment 2. FIG. 6 is a flowchart showing an example of the operation of an occupant state determination device according to embodiment 3. FIG. 7 is a flowchart showing an example of the operation of an occupant state determination device according to embodiment 3.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. FIG. 1 is a diagram showing an example configuration of an occupant state determination device 1A according to embodiment 1. In this embodiment, it is assumed that the occupant state determination device 1A is mounted on a vehicle. However, the occupant state determination device 1A is not limited to being permanently installed in the vehicle, and may be implemented in, for example, a portable device that can be carried into the vehicle, such as a portable navigation device. Furthermore, part or all of the occupant state determination device 1A may be implemented on a server capable of communicating with the vehicle.

[0012] As shown in FIG. 1, the occupant state determination system 100A includes an occupant state determination device 1A, a camera 2A, and a warning device 3A.

[0013] The camera 2A is installed in a position where it can capture the face of the occupant in the driver's seat of the vehicle. The capture range of the camera 2A may be wide enough to capture only the driver's seat, or wide enough to capture the faces of the occupants in all seats of the vehicle.

[0014] The occupant state determination device 1A determines the state of an occupant based on an image of the occupant's face (hereinafter referred to as a "face image") captured by the camera 2A, and outputs the determination result to the warning device 3A. In this embodiment, the occupant state determination device 1A determines the state of the occupant in the driver's seat captured by the camera 2A.

[0015] The warning device 3A issues a warning to the occupant in accordance with the determination result of the occupant state determination unit 15A, and includes a speaker that issues a warning sound or a warning message, a display that displays a warning screen, and the like.

[0016] 1, the occupant state determination device 1A includes a determination unit 10A, a face image acquisition unit 11A, and an eye opening degree calculation unit 12A. The determination unit 10A includes a first eye opening degree determination unit 13A, a second eye opening degree determination unit 14A, an occupant state determination unit 15A, and a warning type determination unit 16A.

[0017] The facial image acquisition unit 11A acquires a facial image of the occupant captured by the camera 2A. The facial image acquisition unit 11A uses a facial recognition technology to extract a facial image of the occupant from the image of the interior of the vehicle captured by the camera 2A. The facial image acquisition unit 11A outputs the facial image of the occupant to the eye opening degree calculation unit 12A.

[0018] The eye opening degree calculation unit 12A detects the highest position of the upper eyelid and the lowest position of the lower eyelid using the facial image of the occupant acquired from the facial image acquisition unit 11A. The eye opening degree calculation unit 12A calculates the distance between the highest position of the upper eyelid and the lowest position of the lower eyelid as the eye opening degree indicating the degree to which the occupant's eyes are open. The eye opening degree calculation unit 12A outputs the calculated eye opening degree to the first eye opening degree determination unit 13A and the second eye opening degree determination unit 14A.

[0019] Although the example in which the eye opening degree calculation unit 12A calculates the distance between the highest point of the upper eyelid and the lowest point of the lower eyelid has been shown, the eye opening degree may be calculated by calculating the distance between the highest point of the upper eyelid and the intersection of a vertical line drawn from the highest point of the upper eyelid and the contour of the lower eyelid. Alternatively, the eye opening degree may be calculated by calculating the distance between the highest point of the upper eyelid and the intersection of a vertical line drawn from the highest point of the upper eyelid to a line connecting the inner corner and outer corner of the eye and the contour of the lower eyelid.

[0020] The first eye-opening degree determination unit 13A determines whether the occupant has their eyes open using the eye-opening degree acquired from the eye-opening degree calculation unit 12A. The first eye-opening degree determination unit 13A determines whether the eye-opening degree is equal to or greater than a first threshold. Specifically, the first eye-opening degree determination unit 13A determines that the occupant has their eyes open when the eye-opening degree is equal to or greater than the first threshold, and determines that the occupant does not have their eyes open when the eye-opening degree is less than the first threshold.

[0021] When the eye opening degree is equal to or greater than the first threshold, the first eye opening degree determination unit 13A outputs a determination result including information indicating that the occupant has their eyes open (hereinafter, information indicating that the occupant has their eyes open is referred to as eye opening state information) to the occupant state determination unit 15A. When the eye opening degree is less than the first threshold, the first eye opening degree determination unit 13A outputs a determination result including information indicating that the occupant does not have their eyes open to the second eye opening degree determination unit 14A.

[0022] When the second eye-opening degree determination unit 14A receives a determination result from the first eye-opening degree determination unit 13A that includes information indicating that the occupant's eyes are not open, the second eye-opening degree determination unit 14A uses the eye-opening degree obtained from the eye-opening degree calculation unit 12A to determine whether the occupant is looking downward or has their eyes closed. Specifically, the second eye-opening degree determination unit 14A determines whether the eye-opening degree is equal to or greater than a second threshold. If the eye-opening degree is equal to or greater than the second threshold, the second eye-opening degree determination unit 14A determines that the occupant is looking downward, and if the eye-opening degree is less than the second threshold, the second eye-opening degree determination unit 14A determines that the occupant is looking downward. Here, the first threshold is greater than the second threshold.

[0023] The second eye opening degree determination unit 14A outputs a determination result to the occupant state determination unit 15A, which includes information indicating that the occupant is in a downward looking state (hereinafter, information indicating that the occupant is in a downward looking state is referred to as downward looking state information), or information indicating that the occupant has their eyes closed (hereinafter, information indicating that the occupant is in a closed eye state is referred to as eye closed state information).

[0024] The occupant state determination unit 15A determines whether the occupant state corresponds to a first occupant state, which is an inattentive state caused by a downward gaze state, using a downward gaze state determination rate, which indicates the proportion of times the second eye-opening degree determination unit 14A determines that the occupant is in a downward gaze state during a first period. Here, the first period is a predetermined period. For example, the first period is a predetermined period (e.g., 20 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h) and the steering angle is within a predetermined range (e.g., ±20 degrees). Note that this is merely an example, and the first period may be appropriately set by an administrator of the occupant state determination device 1A or may be pre-stored in the occupant state determination device 1A.

[0025] The occupant state determination unit 15A will be described in detail below. First, the occupant state determination unit 15A acquires a determination result including eye-open state information from the first eye-opening degree determination unit 13A during a first period, a determination result including downward gaze state information from the second eye-opening degree determination unit 14A during a first period, or a determination result including eye-closed state information from the second eye-opening degree determination unit 14A during a first period.

[0026] Then, the occupant state judgment unit 15A calculates the downward gaze state judgment rate as the proportion of judgment results including downward gaze state information acquired in the first period to judgment results including open eye state information acquired in the first period, judgment results including downward gaze state information acquired in the first period, and judgment results including closed eye state information acquired in the first period.

[0027] The occupant state determination unit 15A then uses the calculated downward gaze state determination rate to determine whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward gaze state. The first occupant state is, for example, a state in which the occupant is operating a smartphone or the like.

[0028] Specifically, the occupant state determination unit 15A determines whether the downward gaze state determination rate is equal to or greater than a third threshold. If the downward gaze state determination rate is equal to or greater than the third threshold, the occupant state determination unit 15A determines that the occupant state is the first state. If the downward gaze state determination rate is less than the third threshold, the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state. Furthermore, even if the occupant state determination unit 15A obtains a determination result including eye openness information from the first eye openness determination unit 13A, the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state. Here, the third threshold is set to, for example, 50 percent. Note that, although an example in which the third threshold is 50 percent has been shown, the third threshold is not limited to this and may be set as appropriate by an administrator of the occupant state determination device 1A, for example.

[0029] The occupant state determination unit 15A outputs a determination result indicating whether the state of the occupant corresponds to the first occupant state to the warning type determination unit 16A.

[0030] Here, the occupant state determination unit 15A determines that the occupant state is the first occupant state when the downward gaze state determination rate is greater than or equal to the third threshold, and explains why the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state when the downward gaze state determination rate is less than the third threshold.

[0031] Generally, when an occupant operates a smartphone or the like, they tend to look downward continuously or frequently. In other words, because the occupant is in a downward gaze state continuously or frequently during a first period, the downward gaze state determination rate tends to be high. Furthermore, because the occupant is in a downward gaze state continuously or frequently during a predetermined period, the occupant spends less time looking forward compared to normal driving, and the occupant can be considered to be in an inattentive state. For these reasons, when the downward gaze state determination rate is equal to or greater than the third threshold, the occupant state determination unit 15A determines that the occupant is in a first occupant state, which is an inattentive state caused by a downward gaze state.

[0032] For the above reasons, the occupant state determination unit 15A determines that the occupant state is the first occupant state when the downward gaze state determination rate is greater than or equal to the third threshold, and determines that the occupant state is an occupant state other than the first occupant state when the downward gaze state determination rate is less than the third threshold.

[0033] The warning type determination unit 16A uses the determination result obtained from the occupant state determination unit 15A to determine the type of warning to be output by the warning device 3A that prompts the occupant to issue a warning. When the occupant state determination unit 15A determines that the occupant state is the first occupant state, the warning type determination unit 16A causes the warning device 3A to output the first warning. When the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state, the warning type determination unit 16A does not cause the warning device 3A to output the first warning.

[0034] 2 is a flowchart showing an example of the operation of the occupant state determination device 1A according to this embodiment. The processing executed by the occupant state determination device 1A according to this embodiment will be described with reference to the flowchart shown in FIG. The operation of the occupant state determination device 1A is started, for example, after the ignition of the vehicle is turned on. It is also preferable to perform the occupant state determination processing every time a face image is acquired from the camera 2A, that is, for each frame.

[0035] First, the face image acquisition unit 11A acquires a face image (step ST101).

[0036] After step ST101, the eye opening degree calculation unit 12A calculates the eye opening degree using the face image acquired from the face image acquisition unit 11A (step ST102).

[0037] After step ST102, the first eye opening degree determination unit 13A determines whether or not the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12A (step ST103).

[0038] If the occupant's eyes are not open (if step ST103 is NO), the second eye opening degree determination unit 14A determines whether the occupant is looking downward or has their eyes closed using the eye opening degree obtained from the eye opening degree calculation unit 12A (step ST104).

[0039] After step ST104, the occupant state judgment unit 15A calculates the downward gaze state judgment rate using the judgment result including the eye open state information acquired from the first eye openness judgment unit 13A, the judgment result including the downward gaze state information acquired from the second eye openness judgment unit 14A, and the judgment result including the eye closed state information acquired from the second eye openness judgment unit 14A (step ST105).

[0040] After step ST105, the occupant state determination unit 15A determines whether the downward gaze state determination rate is equal to or greater than a third threshold value (step ST106).

[0041] If the downward gaze state determination rate is equal to or greater than the third threshold value (YES in step ST106), the occupant state determination unit 15A determines that the state of the occupant is the first occupant state (step ST107).

[0042] If the occupant has their eyes open (if step ST103 is YES), or if the downward gaze state determination rate is less than the third threshold (if step ST106 is NO), the occupant state determination unit 15A determines that the occupant's state is an occupant state other than the first occupant state (step ST108).

[0043] After step ST107, the warning type determining unit 16A causes the warning device 3A to output a first warning (step ST109).

[0044] After step ST108, the warning type determination unit 16A does not cause the warning device 3A to output the first warning (step ST110), and the occupant condition determination device 1A then ends the operation shown in the flowchart of FIG.

[0045] 3 and 4 are diagrams illustrating an example of the hardware configuration of the occupant state determination device 1A according to the first embodiment. The functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A in the occupant state determination device 1A are realized by processing circuits. That is, the occupant state determination device 1A includes a processing circuit for realizing the above functions. The processing circuit may be a processing circuit 100 as dedicated hardware, or a processor 101 that executes a program stored in a memory 102.

[0046] 3 , when the processing circuit is dedicated hardware, the processing circuit 100 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A may be realized by a plurality of processing circuits 100, or the functions of each unit may be realized by a single processing circuit 100.

[0047] 4, when the processing circuit is a processor 101, the functions of the face image acquisition unit 11A, eye opening degree calculation unit 12A, first eye opening degree determination unit 13A, second eye opening degree determination unit 14A, occupant state determination unit 15A, and warning type determination unit 16A are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in memory 102. The processor 101 realizes the function of each unit by reading and executing the program stored in memory 102. In other words, the occupant state determination device 1A includes a memory 102 for storing a program that, when executed by the processor 101, results in the execution of the steps shown in the flowchart of FIG. 2. This program can also be said to cause a computer to execute the procedures or methods of the facial image acquisition unit 11A, eye opening degree calculation unit 12A, first eye opening degree determination unit 13A, second eye opening degree determination unit 14A, occupant state determination unit 15A, and warning type determination unit 16A.

[0048] Here, the processor 101 refers to a CPU (Central Processing Unit), a processing device, an arithmetic unit, a microprocessor, etc. The memory 102 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or a flash memory, or may be a magnetic disk such as a hard disk or a flexible disk, or may be an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc).

[0049] The functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A may be partially implemented by dedicated hardware and partially implemented by software or firmware. In this way, the processing circuit in the occupant state determination device 1A can implement the above-mentioned functions by hardware, software, firmware, or a combination of these.

[0050] The occupant state determination device 1A of this embodiment includes a facial image acquisition unit 11A that acquires a facial image, which is an image of the face of a vehicle occupant; an eye opening degree calculation unit 12A that calculates the eye opening degree indicating the degree to which the occupant's eyes are open using the facial image acquired from the facial image acquisition unit 11A; a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12A; a second eye opening degree determination unit 14A that, when a determination result indicating that the occupant is not in an eye open state is obtained from the first eye opening degree determination unit 13A, determines whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A; and an occupant state determination unit 15A that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward-looking state, using a downward-looking state determination rate that indicates the proportion of times the second eye opening degree determination unit 14A determined that the occupant is in a downward-looking state during a first period. The occupant state determination device 1A can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can reduce the occurrence of erroneous determinations in which an occupant determined to be in a downward-looking state or a closed-eye state is determined to be in the first occupant state even though the occupant is not in the first occupant state, thereby improving the accuracy of occupant state detection.

[0051] The occupant state determination device 1A according to this embodiment includes a face image acquisition unit 11A that acquires a face image of a vehicle occupant's face, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the face image acquired from the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is looking down or has their eyes closed using the eye opening degree acquired from the eye opening degree calculation unit 12A when a determination result indicating that the occupant's eyes are not open is acquired from the first eye opening degree determination unit 13A. and an occupant state determination unit that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward-looking state, by using a downward-looking state determination rate that indicates the proportion of times that the second eye-opening degree determination unit determines that the occupant is in a downward-looking state during a first period of time. The occupant state determination unit determines whether the downward-looking state determination rate is equal to or greater than a third threshold. If the downward-looking state determination rate is equal to or greater than the third threshold, the occupant state determination unit determines that the occupant is in the first occupant state. If the downward-looking state determination rate is less than the third threshold, the occupant state determination unit determines that the occupant is in an occupant state other than the first occupant state. The occupant state determination device can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit. Therefore, the occupant state determination device 1A can suppress the occurrence of erroneous determinations in which an occupant who is determined to be in a downward-looking state or a closed-eye state is determined to be in the first occupant state even though he or she is not in the first occupant state, thereby improving the accuracy of detecting the occupant state.

[0052] The occupant state determination device 1A according to this embodiment includes a face image acquisition unit 11A that acquires a face image of a vehicle occupant's face, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the face image acquired from the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is looking down or has their eyes closed using the eye opening degree acquired from the eye opening degree calculation unit 12A when a determination result indicating that the occupant's eyes are not open is acquired from the first eye opening degree determination unit 13A. The vehicle occupant state determination device 1A includes an occupant state determination unit 15A that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward gaze state, using a downward gaze state determination rate indicating the proportion of times the second eye-opening degree determination unit 14A determined that the occupant is in a downward gaze state during a first period. The vehicle occupant state determination unit 15A further includes a warning type determination unit 16A that determines a type of warning to be output by the warning device 3A to alert the occupant using the determination result obtained from the occupant state determination unit 15A. When the warning type determination unit 16A receives a determination result indicating that the occupant's state is the first occupant state from the occupant state determination unit 15A, the warning type determination unit 16A causes the warning device 3A to output the first warning. With this configuration, the occupant state determination device 1A avoids outputting an unnecessary warning to the occupant by determining that an occupant determined to be in a downward gaze state or a closed eye state is in the first occupant state even though the occupant is in a state other than the first occupant state. Therefore, the occupant state determination device 1A can output a necessary warning to the occupant.

[0053] The occupant state determination system 100A according to this embodiment includes a face image acquisition unit 11A that acquires a face image of a vehicle occupant's face, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the face image acquired from the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is looking down or has their eyes closed using the eye opening degree acquired from the eye opening degree calculation unit 12A when a determination result indicating that the occupant's eyes are not open is acquired from the first eye opening degree determination unit 13A. The abnormal posture determination system includes an occupant state determination device 1A including an occupant state determination unit 15A that determines whether the occupant's state corresponds to a first occupant state, which is an inattention state caused by a downward-looking state, using a downward-looking state determination rate indicating the proportion of times a second eye-opening degree determination unit 14A determined that the occupant is in a downward-looking state during a first period, a camera 2A installed in the vehicle so as to be able to capture at least the occupant's face, and a warning device 3A that outputs a warning to the occupant, in which the occupant state determination device 1A determines the occupant's state using the facial image captured by the camera 2A, and causes the warning device 3A to output a warning according to the determination result of the occupant state determination unit 15A. The occupant state determination system 100A can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of erroneous determinations in which an occupant who is determined to be in a downward-looking state or a closed-eye state is determined to be in the first occupant state even though he or she is not in the first occupant state, thereby improving the accuracy of detecting the occupant state.

[0054] The occupant state determination method according to this embodiment includes the steps of: acquiring a facial image, which is an image of the face of a vehicle occupant; calculating an eye opening degree indicating the degree to which the occupant's eyes are open using the facial image acquired from the facial image acquisition unit 11A, by an eye opening degree calculation unit 12A; determining whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12A; if a determination result indicating that the occupant is not in an eye open state is acquired from the first eye opening degree determination unit 13A, determining whether the occupant is in a downward-looking state or a closed-eye state using the eye opening degree acquired from the eye opening degree calculation unit 12A; and determining whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward-looking state, by an occupant state determination unit 15A, by using a downward-looking state determination rate indicating the proportion of times that the second eye opening degree determination unit 14A has determined that the occupant is in a downward-looking state during a first period. The occupant state determination method can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can reduce the occurrence of erroneous determinations in which an occupant determined to be in a downward-looking state or a closed-eye state is determined to be in the first occupant state even though the occupant is not in the first occupant state, thereby improving the accuracy of occupant state detection.

[0055] The occupant state determination program according to the present embodiment includes a step in which a face image acquisition unit 11A acquires a face image, which is an image obtained by capturing the face of a vehicle occupant; a step in which an eye opening degree calculation unit 12A calculates an eye opening degree indicating the degree of eye opening of the occupant using the face image acquired from the face image acquisition unit 11A; a step in which a first eye opening degree determination unit 13A determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12A; and a step in which a second eye opening degree determination unit 14A determines whether the first eye opening degree determination unit 13A has their eyes open. The occupant state determination program causes the computer to execute the following steps: when a determination result indicating that the occupant does not have their eyes open is acquired from the eye-opening degree calculation unit 12A, determining whether the occupant is in a downward-looking state or a closed-eye state using the eye-opening degree acquired from the eye-opening degree calculation unit 12A; and determining whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward-looking state, using a downward-looking state determination rate indicating the proportion of times the second eye-opening degree determination unit 14A determined that the occupant is in a downward-looking state during a first period. The occupant state determination program can determine whether an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can reduce the occurrence of erroneous determinations in which an occupant determined to be in a downward-looking state or a closed-eye state is determined to be in the first occupant state even though the occupant is not in the first occupant state, thereby improving the accuracy of occupant state detection.

[0056] Embodiment 2. An occupant state determination device 1B in this embodiment will be described. The occupant state determination device 1A in Embodiment 1 is configured to determine the state of an occupant based on the determination result of the eye openness degree and a downward gaze state determination rate calculated from the determination result of the eye openness degree, but the occupant state determination device 1B in this embodiment differs from Embodiment 1 in that it determines the state of an occupant based on the determination result of the eye openness degree and the downward gaze state determination rate calculated from the determination result of the eye openness degree, as well as on the determination result of the eye openness degree and the downward gaze state determination rate calculated from the determination result of the eye openness degree.

[0057] Fig. 5 is a diagram showing an example of the configuration of an occupant state determination device 1B according to this embodiment. As shown in Fig. 5, an occupant state determination system 100B includes a camera 2B, an occupant state determination device 1B, and a camera 2B. The occupant state determination device 1B also includes a determination unit 10B, a face image acquisition unit 11B, and an eye opening degree calculation unit 12B. The determination unit 10B includes a first eye opening degree determination unit 13B, a second eye opening degree determination unit 14B, an occupant state determination unit 15B, and a warning type determination unit 16B.

[0058] In this embodiment, the operation of an occupant state determination unit 15B and the operation of a warning type determination unit 16B are different from those of the occupant state determination device 1A of the first embodiment. The other configurations and operations are the same as those of the first embodiment.

[0059] The occupant state determination unit 15B uses the downward gaze state determination rate and the eye closure state determination rate, which indicates the proportion of times the second eye openness determination unit 14B determines that the occupant has their eyes closed during the first period, to determine whether the occupant's state corresponds to a first occupant state, a second occupant state that is an inattentive state caused by an eye closure state, or a third occupant state that is not an inattentive state. Here, the second occupant state is, for example, a state of drowsiness, a state of drowsy driving, etc. The third occupant state is, for example, a state of checking meters, etc., a state of normal driving, etc.

[0060] The occupant state determination unit 15B will be described in detail below. First, the occupant state determination unit 15B acquires a determination result including eye-open state information from the first eye-opening degree determination unit 13B in the first period, a determination result including downward gaze state information from the second eye-opening degree determination unit 14B in the first period, or a determination result including eye-closed state information from the second eye-opening degree determination unit 14B in the first period.

[0061] Then, the occupant state judgment unit 15B calculates the downward gaze state judgment rate as the proportion of judgment results including downward gaze state information acquired in the first period to judgment results including open eye state information acquired in the first period, judgment results including downward gaze state information acquired in the first period, and judgment results including closed eye state information acquired in the first period.

[0062] In addition, the occupant state determination unit 15B calculates the proportion of determination results including closed-eye state information obtained in the first period to the determination results including open-eye state information obtained in the first period, the determination results including downward-looking state information obtained in the first period, and the determination results including closed-eye state information obtained in the first period as the closed-eye state determination rate.

[0063] Then, the occupant state determination unit 15B uses the downward gaze state determination rate and the closed eye state determination rate to determine whether the occupant state corresponds to the first occupant state, the second occupant state, or the third occupant state.

[0064] Specifically, the occupant state determination unit 15B determines whether the downward gaze state determination rate is equal to or greater than a third threshold. If the downward gaze state determination rate is equal to or greater than the third threshold, the occupant state determination unit 15B determines whether the occupant state is the first state. If the downward gaze state determination rate is less than the third threshold, the occupant state determination unit 15B determines whether the eye-closed state determination rate is equal to or greater than a fourth threshold. If the eye-closed state determination rate is equal to or greater than the fourth threshold, the occupant state determination unit 15B determines the occupant state is the second occupant state. If the eye-closed state determination rate is less than the fourth threshold, the occupant state determination unit 15B determines the occupant state is the third occupant state. Furthermore, even if the occupant state determination unit 15B receives a determination result including eye-open state information from the first eye-opening degree determination unit 13B, the occupant state determination unit 15B determines the occupant state to be the third occupant state. Here, the fourth threshold is set to, for example, 50 percent. Although the fourth threshold value is set to 50% in the above example, it is not limited to this and may be set as appropriate by the manager of the occupant state determination device 1B.

[0065] The occupant state determination unit 15B outputs a determination result indicating whether the occupant state corresponds to the first occupant state, the second occupant state, or the third occupant state to the warning type determination unit 16B.

[0066] Here, the occupant state determination unit 15B determines that the occupant state is the first occupant state when the downward gaze state determination rate is greater than or equal to the third threshold, determines that the occupant state is the second occupant state when the downward gaze state determination rate is less than the third threshold and the closed-eye state determination rate is greater than or equal to the fourth threshold, and determines that the occupant state is the third occupant state when the downward gaze state determination rate is less than the third threshold and the closed-eye state determination rate is less than the fourth threshold.The reasons for this are explained below.

[0067] Generally, when an occupant is drowsy or drowsy while driving, they close their eyes continuously or frequently. Therefore, as described above, the downward gaze state determination rate tends to be lower compared to the first occupant state. Furthermore, because the occupant's eyes are closed continuously or frequently during the first period, the eye-closed state determination rate tends to be higher. Furthermore, because the occupant's eyes are closed continuously or frequently during the predetermined period, the occupant's forward gaze is directed less time than during normal driving, so the occupant can be considered to be in an inattentive state. For these reasons, the occupant state determination unit 15B determines that the occupant's state is the second occupant state when the downward gaze state determination rate is less than the third threshold and the eye-closed state determination rate is equal to or greater than the fourth threshold.

[0068] Furthermore, when the occupant checks a meter or the like, they tend to look downward temporarily or infrequently. Therefore, as described above, the downward gaze state determination rate tends to be lower compared to the first occupant state. Furthermore, because the occupant looks downward temporarily or infrequently during the first period, the closed-eye state determination rate tends to be lower compared to the second occupant state. Furthermore, although the occupant looks downward temporarily or infrequently during the predetermined period, the time spent directing their gaze forward is not reduced compared to the first or second occupant state, so the occupant is considered to be in a non-inattentive state. For these reasons, the occupant state determination unit 15B determines that the occupant's state is the third occupant state if the downward gaze state determination rate is less than the third threshold and the closed-eye state determination rate is less than the fourth threshold.

[0069] For the above reasons, the occupant state determination unit 15B determines that the occupant state is the first occupant state when the downward gaze state determination rate is equal to or greater than the third threshold, determines that the occupant state is the second occupant state when the downward gaze state determination rate is less than the third threshold and the closed-eye state determination rate is equal to or greater than the fourth threshold, and determines that the occupant state is the third occupant state when the downward gaze state determination rate is less than the third threshold and the closed-eye state determination rate is less than the fourth threshold.

[0070] The warning type determination unit 16B uses the determination result obtained from the occupant state determination unit 15B to determine the type of warning to be output by the warning device 3B to alert the occupant. When the occupant state determination unit 15B determines that the occupant's state is the first state, the warning type determination unit 16B causes the warning device 3B to output a first warning. When the occupant state determination unit 15B determines that the occupant's state is the second state, the warning type determination unit 16B causes the warning device 3B to output a second warning that is different from the first warning. When the occupant state determination unit 15B determines that the occupant's state is the third state, the warning type determination unit 16B does not cause the warning device 3B to output the first warning or the second warning.

[0071] 6 is a flowchart showing an example of the operation of the occupant state determination device 1B according to this embodiment. The processing executed by the occupant state determination device 1B according to this embodiment will be described with reference to the flowchart shown in FIG. The operation of the occupant state determination device 1B is started, for example, after the ignition of the vehicle is turned on. It is also preferable to perform the occupant state determination processing every time a face image is acquired from the camera 2B, that is, for each frame.

[0072] First, the face image acquisition unit 11B acquires a face image (step ST201).

[0073] After step ST201, the eye opening degree calculation unit 12B calculates the eye opening degree using the face image acquired from the face image acquisition unit 11B (step ST202).

[0074] After step ST202, the first eye opening degree determination unit 13B determines whether or not the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12B (step ST203).

[0075] If the occupant's eyes are not open (if step ST203 is NO), the second eye opening degree determination unit 14B determines whether the occupant is looking downward or has their eyes closed using the eye opening degree obtained from the eye opening degree calculation unit 12B (step ST204).

[0076] After step ST204, the occupant state judgment unit 15B calculates the downward gaze state judgment rate and the closed eye state judgment rate using the judgment result including the open eye state information acquired from the first eye opening degree judgment unit 13B, the judgment result including the downward gaze state information acquired from the second eye opening degree judgment unit 14B, and the judgment result including the closed eye state information acquired from the second eye opening degree judgment unit 14B (step ST205).

[0077] After step ST205, the occupant state determination unit 15B determines whether the downward gaze state determination rate is equal to or greater than a third threshold value (step ST206).

[0078] If the downward gaze state determination rate is equal to or greater than the third threshold (YES in step ST206), the occupant state determination unit 15B determines that the state of the occupant is the first occupant state (step ST207).

[0079] If the downward gaze state determination rate is less than the third threshold (NO in step ST206), the occupant state determination unit 15B determines whether the eyes-closed state determination rate is equal to or greater than a fourth threshold (step ST208).

[0080] If the eye-closed state determination rate is equal to or greater than the fourth threshold value (YES in step ST208), the occupant state determination unit 15B determines that the state of the occupant is the second occupant state (step ST209).

[0081] If the occupant has their eyes open (if step ST203 is YES), or if the eye-closed state determination rate is less than the fourth threshold (if step ST208 is NO), the occupant state determination unit 15B determines that the occupant's state is the third occupant state (step ST210).

[0082] After step ST207, the warning type determining unit 16B causes the warning device 3B to output a first warning (step ST211).

[0083] After step ST209, the warning type determining unit 16B causes the warning device 3B to output a second warning (step ST212).

[0084] After step ST210, the warning type determination unit 16B does not cause the warning device 3B to output the first warning or the second warning (step ST213). Then, the occupant state determination device 1B ends the operation shown in the flowchart of FIG.

[0085] The occupant state determination device 1B according to this embodiment includes a face image acquisition unit 11B that acquires a face image, which is an image obtained by capturing the face of a vehicle occupant; an eye opening degree calculation unit 12B that calculates an eye opening degree indicating the degree to which the occupant's eyes are open, using the face image acquired from the face image acquisition unit 11B; a first eye opening degree determination unit 13B that determines whether the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12B; and a second eye opening degree determination unit 14B that determines whether the occupant is looking down or has their eyes closed, using the eye opening degree acquired from the eye opening degree calculation unit 12B, when a determination result indicating that the occupant's eyes are not open is acquired from the first eye opening degree determination unit 13B. and an occupant state determination unit that determines whether the state of the occupant corresponds to a first occupant state, which is an inattention state caused by a downward-looking state, by using a downward-looking state determination rate that indicates the proportion of times that the second eye-opening degree determination unit determines that the occupant is in a downward-looking state during the first period. The occupant state determination unit uses an eye-closed state determination rate that indicates the proportion of times that the second eye-opening degree determination unit determines that the occupant is in a closed-eye state during the first period, in addition to the downward-looking state determination rate, to determine whether the state of the occupant corresponds to the first occupant state, a second occupant state, which is an inattention state caused by a closed-eye state, or a third occupant state, which is not an inattention state. With this configuration, in addition to the effects described in the first embodiment, the occupant state determination device 1B can detect the second occupant state and the third occupant state in addition to the first occupant state, thereby being able to detect more occupant states than the occupant state determination device 1A of the first embodiment.

[0086] The occupant state determination device 1B according to this embodiment includes a face image acquisition unit 11B that acquires a face image, which is an image obtained by capturing the face of a vehicle occupant; an eye opening degree calculation unit 12B that calculates an eye opening degree indicating the degree to which the occupant's eyes are open, using the face image acquired from the face image acquisition unit 11B; a first eye opening degree determination unit 13B that determines whether the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12B; a second eye opening degree determination unit 14B that, when a determination result indicating that the occupant is not in an eye opening state is acquired from the first eye opening degree determination unit 13B, determines whether the occupant is in a downward-looking state or a closed-eyed state, using the eye opening degree acquired from the eye opening degree calculation unit 12B; and a percentage of times that the second eye opening degree determination unit 14B has determined that the occupant is in a downward-looking state during a first period. The vehicle driver's seat belt 100 is provided with an occupant state determination unit 15B that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward gaze state, using a downward gaze state determination rate that indicates the first occupant state, and a warning type determination unit 16B that uses the determination result obtained from the occupant state determination unit 15B to determine the type of warning to be output by the warning device 3B to alert the occupant, wherein the warning type determination unit 16B, when obtaining a determination result from the occupant state determination unit 15B indicating that the occupant's state is the first occupant state, causes the warning device 3B to output a first warning, and when obtaining a determination result from the occupant state determination unit 15B indicating that the occupant's state is the second occupant state, causes the warning type determination unit 16B to output a second warning, which is different from the first warning, to the warning device 3B. With this configuration, in addition to the effects described in embodiment 1, the occupant state determination device 1B can output a warning corresponding to the second occupant state in addition to the first occupant state, thereby outputting more warnings corresponding to the occupant state compared to when only a warning corresponding to the first occupant state is output.

[0087] Embodiment 3. An occupant state determination device 1C in this embodiment will be described. The occupant state determination device 1A in Embodiment 1 is configured to determine the state of an occupant based on the determination result of the eye-opening degree and a downward-gaze state determination rate calculated from the determination result of the eye-opening degree, but the occupant state determination device 1C in this embodiment differs from Embodiment 1 in that it determines the state of an occupant based on the determination result of the blink frequency difference in addition to the determination result of the eye-opening degree and the downward-gaze state determination rate calculated from the determination result of the eye-opening degree.

[0088] 7 is a diagram showing an example of the configuration of an occupant state determination device 1C according to an embodiment. As shown in FIG. 7, an occupant state determination system 100C includes a camera 2C, an occupant state determination device 1C, and another camera 2C. The occupant state determination device 1C also includes a determination unit 10C, a face image acquisition unit 11C, an eye opening degree calculation unit 12C, a blink count calculation unit 17C, and a reference blink count estimation unit 18C. The determination unit 10C includes a first eye opening degree determination unit 13C, a second eye opening degree determination unit 14C, a blink count determination unit 19C, an occupant state determination unit 15C, and a warning type determination unit 16C.

[0089] In comparison with occupant state determination device 1A of Embodiment 1, this embodiment further includes a blink count calculation unit 17C, a reference blink count estimation unit 18C, and a blink count determination unit 19C. The following description will focus on blink count calculation unit 17C, reference blink count estimation unit 18C, blink count determination unit 19C, and occupant state determination unit 15C, which has additional functions compared to the configuration of Embodiment 1. The rest of the configuration and operation are the same as those of Embodiment 1.

[0090] The blink count calculation unit 17C calculates a blink count indicating the number of blinks of the occupant during a second period using the eye opening degree acquired from the eye opening degree calculation unit 12C. Here, the second period is a predetermined period. For example, the second period is a predetermined period (e.g., 5 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h) and the steering angle is within a predetermined range (e.g., ±20 degrees). Note that this is merely an example, and the second period may be appropriately set by an administrator of the occupant state determination device 1C or may be pre-stored in the occupant state determination device 1C.

[0091] An example of a method for calculating the number of blinks is shown. The blink count calculation unit 17C determines that the eye is closed when the degree of eye opening is below a second threshold, and determines that the eye is open when the degree of eye opening is above a first threshold that is greater than the second threshold. The blink count calculation unit 17C determines that a blink occurred when an eye is opened within a third period after detecting a closed eye. The blink count calculation unit 17C determines that a blink occurred when an eye is not opened within the third period after detecting a closed eye. In this manner, the blink count calculation unit 17C calculates the number of blinks during the second period. Here, the third period is a predetermined period. For example, the third period is a predetermined period (e.g., 0.2 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h) and the steering angle is within a predetermined range (e.g., ±20 degrees). This is merely an example, and the third period may be set as appropriate by a manager of the occupant state determination device 1C, or may be stored in advance in the occupant state determination device 1C.

[0092] Then, the blink count calculation unit 17C outputs the calculated blink count to the reference blink count estimation unit 18C and the blink count determination unit 19C.

[0093] The reference blink count estimator 18C estimates the reference blink count of the occupant as the reference blink count during a fourth period using the blink count calculated by the blink count calculator 17C. Here, the fourth period is a predetermined period. For example, the fourth period is a predetermined period (e.g., 10 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h) and the steering angle is within a predetermined range (e.g., ±20 degrees). Note that this is merely an example, and the fourth period may be appropriately set by an administrator of the occupant state determination device 1C or may be pre-stored in the occupant state determination device 1C.

[0094] Then, the reference blink count estimation unit 18C outputs the estimated reference blink count to the blink count determination unit 19C.

[0095] The blink count determination unit 19C compares the blink count acquired from the blink count calculation unit 17C with the reference blink count acquired from the reference blink count estimation unit 18C, and determines whether there has been a change in the blink count of the occupant using a blink count difference indicating the difference between the blink count and the reference blink count. Specifically, the blink count determination unit 19C determines whether the blink count difference is equal to or greater than a fifth threshold. If the blink count difference is equal to or greater than the fifth threshold, the blink count determination unit 19C determines that there has been a change in the blink count of the occupant, and if the blink count difference is less than the fifth threshold, it determines that there has been no change in the blink count of the occupant. The fifth threshold may be set as appropriate by an administrator of the occupant state determination device 1C, or may be stored in advance in the occupant state determination device 1C.

[0096] When the blink count difference is less than the fifth threshold, the blink count determination unit 19C outputs a determination result including information indicating that the blink count difference is less than the fifth threshold to the occupant state determination unit 15C. When the blink count difference is equal to or greater than the fifth threshold, the blink count determination unit 19C outputs a determination result including information indicating that the blink count difference is equal to or greater than the fifth threshold to the occupant state determination unit 15C.

[0097] Here, the reason why the occupant state determination device 1C of the present embodiment includes the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C will be explained. Generally, when an occupant operates a smartphone or the like, is drowsy, falls asleep at the wheel, or checks a meter or the like, the number of blinks tends to increase or decrease compared to the number of blinks during normal driving. Therefore, compared to when the first eye-opening degree determination unit 13C determines that the occupant's eyes are not open, if the blink count determination unit 19C further determines that there is a change in the number of blinks of the occupant, it becomes more likely that the occupant is operating a smartphone or the like, is drowsy, falls asleep at the wheel, or is checking a meter or the like. In other words, by further determining whether there is a change in the number of blinks of the occupant by the blink count determination unit 19C, it is possible to more accurately distinguish between a state in which the occupant is operating a smartphone or the like, a state in which the occupant is drowsy, a state in which the occupant is dozing while driving, or a state in which the occupant is checking a meter or the like, and other occupant states.

[0098] When the occupant state determination unit 15C obtains a determination result from the blink count determination unit 19C indicating a change in the number of blinks of the occupant, it determines whether the occupant's state corresponds to the first occupant state using the downward gaze state determination rate, which indicates the percentage of times the second eye opening degree determination unit 14C determined that the occupant was in a downward gaze state during the first period.

[0099] Specifically, the occupant state determination unit 15C determines whether the downward gaze state determination rate is equal to or greater than a third threshold. If the downward gaze state determination rate is equal to or greater than the third threshold, the occupant state determination unit 15C determines that the occupant state is the first state. If the downward gaze state determination rate is less than the third threshold, the occupant state determination unit 15C determines that the occupant state is an occupant state other than the first occupant state. Here, the occupant state other than the first occupant state includes the second occupant state or the third occupant state. Furthermore, the occupant state determination unit 15C determines that the occupant state is an occupant state other than the first occupant state when it receives a determination result including eye open state information from the first eye openness determination unit 13C or a determination result indicating no change in the occupant's blink count from the blink count determination unit 19C.

[0100] The occupant state determination unit 15C outputs a determination result indicating whether the state of the occupant corresponds to the first occupant state to the warning type determination unit 16C.

[0101] 8 is a flowchart showing an example of the operation of the occupant state determination device 1C according to this embodiment. The processing executed by the occupant state determination device 1C according to this embodiment will be described with reference to the flowchart shown in FIG. The operation of the occupant state determination device 1C is started, for example, after the ignition of the vehicle is turned on. It is also preferable to perform the occupant state determination processing every time a face image is acquired from the camera 2C, that is, for each frame.

[0102] First, the face image acquisition unit 11C acquires a face image (step ST301).

[0103] After step ST301, the eye opening degree calculation unit 12C calculates the eye opening degree using the face image acquired from the face image acquisition unit 11C (step ST302).

[0104] After step ST302, the blink count calculation unit 17C calculates the blink count using the degree of eye opening acquired from the eye opening degree calculation unit 12C (step ST303).

[0105] After step ST303, the reference blink count estimation unit 18C estimates the reference blink count using the blink count calculated by the blink count calculation unit 17C (step ST304).

[0106] After step ST304, the first eye opening degree determination unit 13C determines whether or not the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12C (step ST305).

[0107] If the occupant's eyes are not open (if step ST305 is NO), the second eye opening degree determination unit 14C determines whether the occupant is looking downward or has their eyes closed using the eye opening degree obtained from the eye opening degree calculation unit 12C (step ST306).

[0108] After step ST306, the blink count determination unit 19C determines whether or not there is a change in the blink count of the occupant (step ST307).

[0109] If there is a change in the number of blinks of the occupant (if step ST307 is YES), the occupant state judgment unit 15C calculates the downward gaze state judgment rate using the judgment result including the eye open state information acquired from the first eye openness judgment unit 13C, the judgment result including the downward gaze state information acquired from the second eye openness judgment unit 14C, and the judgment result including the eye closed state information acquired from the second eye openness judgment unit 14C (step ST308).

[0110] After step ST308, the occupant state determination unit 15C determines whether the downward gaze state determination rate is equal to or greater than a third threshold value (step ST309).

[0111] If the downward gaze state determination rate is equal to or greater than the third threshold value (YES in step ST309), the occupant state determination unit 15C determines that the state of the occupant is the first occupant state (step ST310).

[0112] If the occupant has their eyes open (if step ST305 is YES), if there is no change in the number of times the occupant blinks (if step ST307 is NO), or if the downward gaze state determination rate is less than the third threshold (if step ST309 is NO), the occupant state determination unit 15C determines that the occupant's state is an occupant state other than the first occupant state (step ST311).

[0113] After step ST310, the warning type determining unit 16C causes the warning device 3C to output a first warning (step ST312).

[0114] After step ST311, the warning type determination unit 16C does not cause the warning device 3C to output the first warning (step ST313).Then, the occupant state determination device 1C ends the operation shown in the flowchart of FIG.

[0115] The occupant state determination device 1C according to a modified example of the present embodiment includes a face image acquisition unit 11C that acquires a face image of a vehicle occupant's face, an eye opening degree calculation unit 12C that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the face image acquired from the face image acquisition unit 11C, a first eye opening degree determination unit 13C that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12C, a second eye opening degree determination unit 14C that, when a determination result indicating that the occupant is not in an eye open state is acquired from the first eye opening degree determination unit 13C, determines whether the occupant is in a downward-looking state or a closed-eyed state using the eye opening degree acquired from the eye opening degree calculation unit 12C, and a ratio of times during a first period when the second eye opening degree determination unit 14C determines that the occupant is in a downward-looking state. The occupant state determination unit 15C is provided with an occupant state determination rate that indicates whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward gaze state, using the downward gaze state determination rate that indicates the downward gaze state, and further with a blink count determination unit 19C that determines whether there has been a change in the blink count that indicates the number of times the occupant blinks.When the occupant state determination unit 15C obtains a determination result from the blink count determination unit 19C that indicates a change in the number of blinks of the occupant, the occupant state determination unit 15C uses the downward gaze state determination rate to determine whether the occupant's state corresponds to the first occupant state, and when the occupant state determination unit 15C obtains a determination result from the blink count determination unit 19C that indicates no change in the number of blinks of the occupant, the occupant state determination unit 15C does not use the downward gaze state determination rate to determine whether the occupant's state corresponds to the first occupant state. With this configuration, the occupant state determination device 1C can more accurately distinguish between a state in which the occupant is operating a smartphone or the like, a state in which the occupant is drowsy, a state in which the occupant is dozing off while driving, or a state in which the occupant is checking a meter or the like, and other occupant states. As a result, the occupant state determination device 1C can further improve the occupant state detection accuracy compared to the occupant state determination device 1A of the first embodiment.

[0116] In this embodiment, the combination of the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C, which are configurations unique to this embodiment, with embodiment 1 has been described. However, the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C, which are configurations unique to this embodiment, may also be combined with embodiment 2.

[0117] 1A Occupant state determination device, 2A Camera, 3A Warning device, 10A Determination unit, 11A Face image acquisition unit, 12A Eye opening degree calculation unit, 13A First eye opening degree determination unit, 14A Second eye opening degree determination unit, 15A Occupant state determination unit, 16A Warning type determination unit, 100A Occupant state determination system

Claims

1. a face image acquisition unit that acquires a face image that is an image obtained by capturing a face of a vehicle occupant; an eye opening degree calculation unit that calculates an eye opening degree indicating a degree of opening of the eyes of the occupant by using the face image acquired from the face image acquisition unit; a first eye opening degree determination unit that determines whether the occupant has their eyes open by using the eye opening degree acquired from the eye opening degree calculation unit; a second eye opening degree determination unit that, when a determination result indicating that the occupant does not have their eyes open is obtained from the first eye opening degree determination unit, determines whether the occupant is looking downward or has their eyes closed using the eye opening degree obtained from the eye opening degree calculation unit; an occupant state determination unit that determines whether the state of the occupant corresponds to a first occupant state, which is an inattention state caused by the downward gaze state, by using a downward gaze state determination rate that indicates a rate at which the second eye opening determination unit determines that the occupant is in the downward gaze state during a first period; An occupant state determination device comprising:

2. The degree of eye opening is a distance between the highest point of an upper eyelid detected from the face image and the lowest point of a lower eyelid detected from the face image. The occupant condition determination device according to claim 1 .

3. The first eye opening degree determination unit determines whether the eye opening degree is equal to or greater than a first threshold value, the first eye open degree determination unit determines that the occupant has an eye open state when the eye open degree is equal to or greater than the first threshold value, and determines that the occupant does not have an eye open state when the eye open degree is less than the first threshold value, The second eye opening degree determination unit determines whether the eye opening degree is equal to or greater than a second threshold value, the second eye opening degree determination unit determines that the occupant is in the downward gaze state when the eye opening degree is equal to or greater than the second threshold, and determines that the occupant is in the eye closing state when the eye opening degree is less than the second threshold, The first threshold is greater than the second threshold. The occupant condition determination device according to claim 1 .

4. The occupant state determination unit determines whether the downward gaze state determination rate is equal to or greater than a third threshold, the occupant state determination unit determines that the state of the occupant is the first occupant state when the downward gaze state determination rate is equal to or greater than the third threshold value, the occupant state determination unit determines that the state of the occupant is an occupant state other than the first occupant state when the downward gaze state determination rate is less than the third threshold value. The occupant state determination device according to any one of claims 1 to 3.

5. the occupant state determination unit uses, in addition to the downward gaze state determination rate, an eye closed state determination rate indicating the proportion of times that the second eye openness determination unit has determined that the occupant is in the eye closed state during the first period to determine whether the state of the occupant corresponds to any one of the first occupant state, a second occupant state which is an inattention state caused by the eye closed state, and a third occupant state which is not the inattention state. The occupant condition determination device according to claim 1 .

6. The occupant state determination unit determines whether the downward gaze state determination rate is equal to or greater than a third threshold, the occupant state determination unit determines that the state of the occupant is the first occupant state when the downward gaze state determination rate is equal to or greater than the third threshold value, When the downward gaze state determination rate is less than the third threshold, the occupant state determination unit determines whether the eye closed state determination rate is equal to or greater than a fourth threshold, the occupant state determination unit determines that the state of the occupant is the second occupant state when the eye-closed state determination rate is equal to or greater than the fourth threshold value, the occupant state determination unit determines that the state of the occupant is the third occupant state when the eye-closed state determination rate is less than the fourth threshold value. The occupant condition determining device according to claim 5.

7. a warning type determination unit that determines a type of warning to be output by a warning device that prompts the occupant to issue a warning using a determination result acquired from the occupant state determination unit, the warning type determination unit, when receiving a determination result indicating that the state of the occupant is the first occupant state from the occupant state determination unit, causes the warning device to output a first warning. The occupant condition determination device according to claim 1 or 2.

8. a warning type determination unit that determines a type of warning to be output by a warning device that prompts the occupant to issue a warning using a determination result acquired from the occupant state determination unit, the warning type determination unit, when receiving a determination result indicating that the state of the occupant is the first occupant state from the occupant state determination unit, causes the warning device to output a first warning; the warning type determination unit, when acquiring a determination result indicating that the state of the occupant is the second occupant state from the occupant state determination unit, causes the warning device to output a second warning different from the first warning. The occupant condition determining device according to claim 5.

9. a blinking frequency determination unit that determines whether or not there is a change in the blinking frequency of the occupant; Equipped with the occupant state determination unit, when receiving a determination result indicating that there is a change in the number of blinks of the occupant from the blink count determination unit, determines whether the state of the occupant corresponds to the first occupant state by using the downward gaze state determination rate; and when the occupant state determination unit obtains a determination result indicating that there is no change in the number of blinks of the occupant from the blink count determination unit, the occupant state determination unit does not determine whether the state of the occupant corresponds to the first occupant state by using the downward gaze state determination rate. The occupant condition determination device according to claim 1 .

10. a blinking frequency determination unit that determines whether or not there is a change in the blinking frequency of the occupant; Equipped with the occupant state determination unit, when acquiring a determination result indicating that there is a change in the number of blinks of the occupant from the blink count determination unit, determines whether the state of the occupant corresponds to any one of the first occupant state, the second occupant state, and the third occupant state, using the downward gaze state determination rate and the closed-eye state determination rate; when the occupant state determination unit acquires a determination result indicating that there is no change in the number of blinks of the occupant from the blink count determination unit, the occupant state determination unit does not determine whether the state of the occupant corresponds to any one of the first occupant state, the second occupant state, and the third occupant state, using the downward gaze state determination rate and the closed-eye state determination rate. The occupant condition determining device according to claim 5.

11. a blinking frequency calculation unit that calculates the blinking frequency during a second period by using the eye opening degree acquired from the eye opening degree calculation unit; a reference blink count estimation unit that estimates a reference blink count of the occupant as a reference blink count by using the blink count calculated by the blink count calculation unit during a fourth period; Equipped with the blink count determination unit compares the blink count acquired from the blink count calculation unit with the reference blink count acquired from the reference blink count estimation unit, and determines whether or not there is a change in the blink count by using a blink count difference indicating a difference between the blink count and the reference blink count. The occupant condition determining device according to claim 9.

12. The blink count determination unit determines whether the blink count difference is equal to or greater than a fifth threshold, the blink count determination unit determines that there is a change in the blink count of the occupant when the blink count difference is equal to or greater than the fifth threshold, and determines that there is no change in the blink count of the occupant when the blink count difference is less than the fifth threshold. The occupant condition determination device according to claim 11.

13. An abnormal posture determination system including the occupant state determination device according to any one of claims 1 to 3, 5, 6, and 8 to 12, a camera installed in the vehicle so as to be able to photograph at least a face of the occupant, and a warning device that outputs a warning to the occupant, the occupant state determination device determines the state of the occupant using a face image captured by the camera, and causes the warning device to output a warning according to a determination result of the occupant state determination unit.

14. An occupant state determination method executed by an occupant state determination device, comprising: A facial image acquisition unit acquires a facial image of a vehicle occupant, the facial image being an image of the face of the vehicle occupant. An eye opening degree calculation unit calculates an eye opening degree indicating a degree of opening of the eyes of the occupant by using the facial image acquired from the facial image acquisition unit. a step of a first eye open degree determination unit determining whether or not the occupant has their eyes open by using the eye open degree acquired from the eye open degree calculation unit; a step of a second eye opening degree determination unit determining whether the occupant is looking downward or has their eyes closed using the eye opening degree obtained from the eye opening degree calculation unit when the second eye opening degree determination unit obtains a determination result indicating that the occupant is not in an eye open state from the first eye opening degree determination unit; an occupant state determination unit determining whether the state of the occupant corresponds to a first occupant state, which is an inattention state caused by the downward gaze state, by using a downward gaze state determination rate indicating a rate at which the second eye opening determination unit determines that the occupant is in the downward gaze state during a first period; An occupant state determination method comprising:

15. A facial image acquisition unit acquires a facial image of a vehicle occupant, the facial image being an image of the face of the vehicle occupant. An eye opening degree calculation unit calculates an eye opening degree indicating a degree of opening of the eyes of the occupant by using the facial image acquired from the facial image acquisition unit. a step of a first eye open degree determination unit determining whether or not the occupant has their eyes open by using the eye open degree acquired from the eye open degree calculation unit; a step of a second eye opening degree determination unit determining whether the occupant is looking downward or has their eyes closed using the eye opening degree obtained from the eye opening degree calculation unit when the second eye opening degree determination unit obtains a determination result indicating that the occupant is not in an eye open state from the first eye opening degree determination unit; an occupant state determination unit determining whether the state of the occupant corresponds to a first occupant state, which is an inattention state caused by the downward gaze state, by using a downward gaze state determination rate indicating a rate at which the second eye opening determination unit determines that the occupant is in the downward gaze state during a first period; An occupant state determination program for causing a computer to execute the above.