Occupant state detection device, occupant state detection program, and occupant state detection method

Abstract

An occupant state detection device (100, 200, 300, 400, 500) comprises: an image information acquisition unit (13) that acquires image information obtained by capturing an image of the inside of a vehicle compartment (R); an eyelid opening degree detection unit (14) that detects the eyelid opening degree of an occupant (J2) in the vehicle compartment (R) on the basis of the image information acquired by the image information acquisition unit (13); an alertness level estimation unit (16) that estimates the alertness level of the occupant (J2); and an inattentiveness detection unit (19) that detects inattentiveness of the occupant (J2), on the basis of the eyelid opening degree of the occupant (J2) detected by the eyelid opening degree detection unit (14) and the alertness level of the occupant (J2) estimated by the alertness level estimation unit (16).

Description

Occupant condition detection device, occupant condition detection program, and occupant condition detection method

[0001] The present disclosure relates to an occupant state detection device, an occupant state detection program, and an occupant state detection method.

[0002] Conventionally, an occupant status monitoring device has been disclosed that detects the gaze of an occupant in a vehicle from an image obtained by photographing the eyeballs of the occupant and determines whether the occupant is looking away based on the detected gaze (see, for example, Patent Document 1).

[0003] JP 2023-139930 A

[0004] However, with the conventional technology, there is a problem in that if the line of sight of the occupant cannot be detected, it is not possible to detect that the occupant is looking away.

[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to provide an occupant status detection device, an occupant status detection program, and an occupant status detection method that can detect an occupant's inattentiveness even when the occupant's line of sight cannot be detected.

[0006] The occupant status detection device according to the present disclosure is characterized by including an image information acquisition unit that acquires image information obtained by imaging the interior of the vehicle cabin, an eyelid opening detection unit that detects the degree of eyelid opening of an occupant in the vehicle cabin based on the image information acquired by the image information acquisition unit, an alertness estimation unit that estimates the alertness of the occupant, and an inattentiveness detection unit that detects whether the occupant is looking away based on the eyelid opening degree of the occupant detected by the eyelid opening detection unit and the alertness of the occupant estimated by the alertness estimation unit.

[0007] According to the present disclosure, the occupant's inattentiveness is detected based on the degree of eyelid opening and the occupant's level of alertness, so that the occupant's inattentiveness can be detected even when the occupant's line of sight cannot be detected.

[0008] FIG. 1 is a side view showing a vehicle according to a first embodiment. FIG. 2 is a block diagram showing a schematic configuration of an occupant state detection device according to the first embodiment. FIG. 3 is a block diagram showing an example of a hardware configuration of an occupant state detection device according to the first embodiment. FIG. 4 is a block diagram showing an example of a hardware configuration of an occupant state detection device according to the first embodiment. FIG. 5 is a block diagram showing a schematic configuration of an occupant state detection device according to a second embodiment. FIG. 6 is a flowchart showing a process performed by an occupant state detection device according to the second embodiment. FIG. 7 is a block diagram showing a schematic configuration of an occupant state detection device according to a third embodiment. FIG. 8 is a flowchart showing a process performed by an occupant state detection device according to the third embodiment. FIG. 9 is a block diagram showing a schematic configuration of an occupant state detection device according to a fourth embodiment. FIG. 10 is a flowchart showing a process performed by an occupant state detection device according to the fourth embodiment. FIG. 11 is a block diagram showing a schematic configuration of an occupant state detection device according to a fifth embodiment. FIG. 12 is a flowchart showing a process performed by an occupant state detection device according to the fifth embodiment.

[0009] Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the drawings. Embodiment 1. First, a schematic configuration of a vehicle 1 according to embodiment 1 will be described with reference to FIGS. 1 and 2. FIG. 1 is a side view showing the vehicle 1 according to embodiment 1. The vehicle 1 includes a seat 3 arranged in a passenger compartment R defined by a main body of the vehicle 1, an imaging device 110 that images the interior of the passenger compartment R, an occupant status detection device 100 that detects the status of an occupant J2, and an alarm device that issues an alarm based on an input signal from the occupant status detection device 100. For example, the vehicle 1 includes a plurality of seats 3, including a driver's seat 2 in which a driver J1 of the vehicle 1, who serves as the occupant J2, is seated.

[0010] In the first embodiment, the direction opposite to the Z direction shown in Fig. 1, which is the direction in which the vehicle 1 moves when the steering wheel (not shown) of the vehicle 1 is in the neutral position and the gear position of the transmission (not shown) of the vehicle 1 is in the forward position, is defined as the front of the vehicle, and the front, rear, left and right of the vehicle are defined based on this. Also, in the first embodiment, the front of the vehicle is also simply referred to as the "forward".

[0011] The imaging device 110 captures an image of the interior of the vehicle compartment R while the occupant J2 is seated in the seat 3, thereby acquiring image information of the interior of the vehicle compartment R. For example, the imaging device 110 has an imaging element (image sensor) and a lens (not shown), and converts light captured through the lens into a signal using the imaging element. The imaging element is, for example, a solid-state imaging element such as a charge coupled device (CCD) image sensor or a complete metal oxide semiconductor (CMOS) image sensor.

[0012] The imaging device 110 captures an image of an area including the face of an occupant J2 seated in one of the seats 3 in the vehicle cabin R. For example, the imaging device 110 is disposed in a position where it can capture an image of a driver J1 seated in the driver's seat 2 with his / her face facing forward and his / her eyelids open, and captures an image of an area including at least the face of the driver J1. Specifically, the imaging device 110 is disposed in front of the driver J1 seated in the driver's seat 2 and slightly below the face of the driver J1, or in front of the driver J1 seated in the driver's seat 2 and above the face of the driver J1, and captures an image of an area including the face of the driver J1. For example, the imaging device 110 is disposed on the dashboard or an overhead console and captures an image of an area including the face of the driver J1. Furthermore, for example, the imaging device 110 captures an image of the vehicle cabin R at a preset frame rate. The imaging device 110 converts image information obtained by capturing an image into an electrical signal and outputs the signal.

[0013] The alarm device 120 alerts the driver of information relating to the detection result by the occupant state detection device 100 based on an input signal from the occupant state detection device 100. For example, the alarm device 120 is configured to include an audio output device such as a speaker that alerts the driver of information by outputting sound based on the input signal from the occupant state detection device 100, a display device such as a liquid crystal monitor that alerts the driver of information by displaying an image based on the input signal from the occupant state detection device 100, and a light-emitting device such as an LED that alerts the driver of information by changing its light-emitting mode based on the input signal from the occupant state detection device 100. For example, when the occupant state detection device 100 detects that the occupant J2 is looking aside, the alarm device 120 alerts the driver of information by outputting a message urging the occupant J2 not to look aside based on the input signal from the occupant state detection device 100 that the occupant J2 is looking aside. Furthermore, for example, the alarm device 120 calculates the length of continuous inattentive time, which is the time during which the inattentive state of occupant J2 is continuously maintained, based on an input signal from the occupant state detection device 100 indicating that occupant J2 is in a state of looking away, and if the calculated length of continuous inattentive time is equal to or greater than a preset threshold, it issues a message to urge occupant J2 not to look away.

[0014] The occupant state detection device 100 detects the state of the occupant J2 in the vehicle compartment R based on an input signal from the imaging device 110. For example, based on the input signal from the imaging device 110, the occupant state detection device 100 detects whether the occupant J2 is looking aside as the state of the occupant J2 in the vehicle compartment R. Specifically, based on the input signal from the imaging device 110, the occupant state detection device 100 detects whether the driver J1, who is seated in the driver's seat 2 in the vehicle compartment R, is looking aside as the state of the driver J1. In other words, the occupant state detection device 100 detects whether the driver J1 is looking aside based on image information obtained by capturing an image of the inside of the vehicle compartment R by the imaging device 110. Note that the following description will focus on the occupant state detection device 100 detecting the state of the driver J1 as the occupant J2, but the occupant state detection device 100 may detect the state of any occupant J2 in the vehicle 1, rather than being limited to the driver J1. The occupant state detection device 100 outputs the detection result to the notification device 120. In the first embodiment, "looking aside" refers to a state in which the occupant J2 is looking at anything other than the area ahead of the vehicle 1 and surrounding areas of the road ahead.

[0015] 2 is a block diagram showing a schematic configuration of an occupant state detection device 100 according to Embodiment 1. The occupant state detection device 100 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an alertness estimation unit 16, and an inattentive driving detection unit 19, and is electrically connected to an image capture device 110 and a notification device 120 so as to be able to communicate with them. The input unit 11 accepts input signals from the image capture device 110 and various devices of the vehicle 1 (not shown). The output unit 12 outputs a signal to the notification device 120 according to the result of processing by the occupant state detection device 100.

[0016] The image information acquisition unit 13 acquires image information obtained by imaging the inside of the vehicle compartment R based on an input signal from the imaging device 110. For example, the image information acquisition unit 13 acquires image information including an image of the face of the driver J1 obtained by imaging the inside of the vehicle compartment R. Specifically, the image information acquisition unit 13 acquires image information including an image of the face of the driver J1 obtained by imaging the inside of the vehicle compartment R.

[0017] The eyelid opening degree detection unit 14 detects the degree of eyelid opening of the driver J1 in the vehicle compartment R based on the image information acquired by the image information acquisition unit 13. For example, the eyelid opening degree detection unit 14 detects the degree of eyelid opening of the driver J1 in the vehicle compartment R based on the image information acquired by the image information acquisition unit 13. Furthermore, for example, the eyelid opening degree detection unit 14 detects the upper eyelid and the lower eyelid of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a known image recognition technology. For example, the eyelid opening degree detection unit 14 extracts feature points indicating the upper eyelid and the lower eyelid of the driver J1 that are present in the face detection area of ​​the image information acquired by the image information acquisition unit 13 by using a known edge detection technology in a state where a face detection area, which is an area where the face of the driver J1 may be present, is set in advance in the image information acquired by the image information acquisition unit 13. Next, the eyelid opening degree detection unit 14 calculates, for example, the maximum value of the eyelid distance, which is the vertical distance between the upper eyelid and the lower eyelid of the occupant in the image information.The eyelid opening degree detection unit 14 then detects the eyelid opening degree by calculating, for example, the ratio of the maximum value of the eyelid distance to a preset reference value.For example, such a reference value of the eyelid opening degree is a value set based on the average eyelid distance of an adult of average build when awake.

[0018] The reference value for the eyelid opening degree is not limited to a value set based on the average eyelid distance of an adult of average build when awake. The reference value for the eyelid opening degree may be any value that can determine the degree of eyelid opening of the driver J1 by comparing it with the eyelid distance acquired by the eyelid opening degree detection unit 14. For example, the reference value for the eyelid opening degree may be a value set based on the average eyelid distance during a predetermined period during which the driver J1 in the vehicle 1 is awake and facing forward, or may be a value set based on the eyelid distance of occupants who have previously ridden the vehicle 1. Furthermore, the eyelid opening degree is not limited to the ratio of the maximum eyelid distance to a predetermined reference value. The eyelid opening degree may be any value that quantifies the degree of eyelid opening of the occupant. For example, the reference value for the eyelid opening degree may be the maximum eyelid distance itself, the ratio of the eyelid distance to the maximum distance between the left and right eyelids, or the area of ​​the eyeball shown in the image information.

[0019] The arousal level estimation unit 16 estimates the arousal level of the driver J1 based on various information. Specifically, the arousal level estimation unit 16 estimates the arousal level of the driver J1 based on image information acquired by the image information acquisition unit 13. For example, the arousal level estimation unit 16 estimates the arousal level of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a trained model that learns using learning data that associates image information obtained by capturing an area including at least the face of the occupant with the arousal level, and outputs the arousal level of the occupant in response to input of image information including an image of the occupant's face. Note that the arousal level estimation unit 16 is preferably configured to estimate the arousal level of the driver J1 based on the trained model that learns using the image information acquired by the image information acquisition unit 13 and learning data that associates image information obtained by capturing an area including the face of the occupant and part or all of the upper body excluding the face with the arousal level, and outputs the arousal level of the occupant in response to input of image information including the face of the occupant and part or all of the upper body excluding the face.

[0020] Furthermore, for example, the arousal level estimation unit 16 estimates the arousal level of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a known image recognition technique. Specifically, the arousal level estimation unit 16 extracts feature points of each body part of the driver J1 present in a detection area of ​​the image information acquired by the image information acquisition unit 13, the detection area being an area where the driver J1 may be present, by using a known edge detection technique in a state where the detection area is set in advance. Next, the arousal level estimation unit 16 estimates the arousal level of the driver J1 based on the extracted feature points of each body part of the driver J1.

[0021] For example, the arousal level estimation unit 16 estimates the arousal level by detecting body movements of the driver J1 attempting to resist drowsiness based on the extracted feature points of each body part of the driver J1. For example, the arousal level estimation unit 16 estimates the arousal level based on an increase or decrease in the body movements of the driver J1 attempting to resist drowsiness. Furthermore, for example, the arousal level estimation unit 16 estimates the arousal level by detecting the frequency of characteristic movements of the driver J1, such as stretching, slapping the body, touching the face or head, etc. Generally, when a person attempts to resist drowsiness, their body movements increase and they make characteristic movements, such as stretching, slapping the body, and touching the face or head.

[0022] It is desirable that the wakefulness estimation unit 16 is configured to estimate the wakefulness of the occupant based on factors other than at least the degree of eyelid opening of the occupant. In other words, it is desirable that the wakefulness estimation unit 16 is configured so that the estimation result of the wakefulness of the occupant is not linked to the detection result of the eyelid opening of the occupant by the eyelid opening detection unit 14. It is desirable that the wakefulness estimation unit 16 is configured to estimate the wakefulness of the occupant based on factors other than at least the degree of eyelid opening of the occupant. This does not mean that the degree of eyelid opening has no effect on the estimation result of the wakefulness of the occupant, and the degree of eyelid opening of the occupant may be included as one of the factors when estimating the wakefulness of the occupant.

[0023] Furthermore, for example, the arousal level estimation unit 16 estimates the arousal level by detecting body movements that indicate drowsiness of the driver J1 based on the extracted feature points of each part of the driver J1. For example, the arousal level estimation unit 16 estimates the arousal level by detecting the mouth opening degree of the driver J1. For example, the arousal level estimation unit 16 estimates the arousal level by detecting the frequency of yawning of the driver J1. For example, the arousal level estimation unit 16 estimates the arousal level by detecting the frequency of blinking of the driver J1. For example, the arousal level estimation unit 16 estimates the arousal level by detecting a change in the line of sight of the driver J1. Generally, when a person is drowsy, the mouth opening degree, frequency of yawning, and frequency of blinking increase, and change in line of sight decreases.

[0024] The inattentive driving detection unit 19 detects inattentive driving by the driver J1 based on the eyelid opening degree of the driver J1 detected by the eyelid opening degree detection unit 14 and the alertness of the driver J1 estimated by the alertness degree estimation unit 16. For example, the inattentive driving detection unit 19 compares the eyelid opening degree of the driver J1 detected by the eyelid opening degree detection unit 14 with a preset eyelid opening degree threshold, and compares the alertness of the driver J1 estimated by the alertness degree estimation unit 16 with a preset alertness threshold, and detects inattentive driving by the driver J1 based on these comparison results. Details of the processing performed by the inattentive driving detection unit 19 will be described later.

[0025] Next, the hardware configuration of the occupant state detection device 100 will be described with reference to Figures 3 and 4. Figure 3 is a diagram showing an example of the hardware configuration of the occupant state detection device 100, and Figure 4 is a diagram showing an example of the hardware configuration of the occupant state detection device 100 that is different from that shown in Figure 3. For example, as shown in Figure 3, the occupant state detection device 100 is configured as a computer having a processor 100a, a memory 100b, and an I / O port 100c, and is configured so that the processor 100a reads and executes a program stored in the memory 100b.

[0026] 4, the occupant status detection device 100 is configured as a computer having a processing circuit 100d, which is dedicated hardware, and an I / O port 100c. The processing circuit 100d is configured, for example, by a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each function of the occupant status detection device 100 is realized by the processor 100a or the processing circuit 100d, which is dedicated hardware, executing a program. Note that the occupant status detection device 100 may include hardware other than the above, such as a hardware timer used for timing.

[0027] Next, the processing performed by the occupant state detection device 100 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the processing performed by the occupant state detection device 100. As shown in Fig. 5, when the occupant state detection device 100 starts the processing, it first acquires image information obtained by capturing an image of the interior of the vehicle compartment R (step ST1). In this processing, the occupant state detection device 100 acquires image information obtained by the imaging device 110 capturing an image of the interior of the vehicle compartment R using the image information acquisition unit 13.

[0028] After the process of step ST1 is performed, the occupant state detection device 100 then detects the eyelid opening degree of the driver J1 (step ST2). In this process, the occupant state detection device 100 detects the eyelid opening degree of the driver J1 by the eyelid opening degree detection unit 14 based on the image information acquired by the image information acquisition unit 13.

[0029] After the process of step ST2, the occupant state detection device 100 next estimates the alertness level of the driver J1 (step ST3). In this process, the occupant state detection device 100 estimates the alertness level of the driver J1 by the alertness level estimation unit 16 based on, for example, image information acquired by the image information acquisition unit 13.

[0030] After the processing of step ST3, the occupant state detection device 100 next determines whether the degree of eyelid opening of the driver J1 is equal to or greater than a preset threshold (step ST6). Generally, when detecting the degree of eyelid opening using image recognition technology based on image information obtained when capturing an image of the occupant's face, the eyelid opening degree tends to be detected as a lower value when the occupant is looking aside and facing downward rather than forward, such as when the occupant is operating a device located on the dashboard of the vehicle or when the occupant is operating a mobile terminal, than when the occupant is facing forward. In the processing of step ST6, the occupant state detection device 100, for example, compares the eyelid opening degree of the driver J1 with a threshold value for the eyelid opening degree that is considered to be highly likely to indicate that the driver J1 is facing forward and is awake, thereby determining, via the inattentiveness detection unit 19, whether the eyelid opening degree of the driver J1 is a sufficient value to indicate that the driver J1 is not looking aside.

[0031] If the degree of eyelid opening of the driver J1 is less than the threshold in the process of step ST6 (NO in step ST6), the occupant state detection device 100 then determines whether the alertness of the driver J1 is equal to or greater than a preset threshold of alertness (step ST9). In this process, the occupant state detection device 100 determines whether the driver J1 is in a state of drowsiness by using the inattentiveness detection unit 19, for example, by comparing the alertness of the driver J1 with a threshold of alertness that is considered sufficient to determine that the driver J1 is not drowsy.

[0032] If the wakefulness level of the driver J1 is equal to or higher than the threshold in the process of step ST9 (YES in step ST9), the occupant state detection device 100 then detects that the driver J1 is looking aside (step ST12). In other words, based on the fact that the wakefulness level of the driver J1 is equal to or higher than the threshold in the process of step ST9, the occupant state detection device 100 determines that the driver J1 is looking aside. As described above, when the occupant is looking aside, with their face facing downward rather than forward, the degree of eyelid opening tends to be detected as a lower value than when the occupant is facing forward. Furthermore, generally, when an occupant sitting in the seat 3 falls asleep, they tend to turn their face downward rather than forward. Therefore, when determining the state of the occupant based solely on the degree of eyelid opening, it is difficult to determine whether the occupant is looking aside with their face facing downward rather than forward, or whether the occupant is dozing with their eyelids closed. The occupant state detection device 100 according to the first embodiment detects that the driver J1 is not dozing but is looking aside with his / her head turned downward rather than forward, based on the fact that the level of alertness is equal to or greater than a threshold when the degree of eyelid opening is lower than a threshold.

[0033] After performing the process of step ST12, the occupant status detection device 100 then outputs the detection result of the driver J1's inattentiveness to the alarm device 120 (step ST13). If the eyelid opening degree of the driver J1 is equal to or greater than the threshold value in the process of step ST6 (YES in step ST6), if the wakefulness level of the driver J1 is less than the threshold value in the process of step ST9 (NO in step ST9), or if the process of step ST13 is performed, the occupant status detection device 100 ends the process. Note that the occupant status detection device 100 may be configured to output a detection result indicating that the driver J1's inattentiveness was not detected to the alarm device 120 when the occupant status detection device 100 has not detected the driver J1's inattentiveness.

[0034] As described above, the occupant state detection device 100 according to the first embodiment is characterized by including an image information acquisition unit 13 that acquires image information obtained by imaging the interior of the vehicle compartment R, an eyelid opening detection unit 14 that detects the degree of eyelid opening of the driver J1 in the vehicle compartment R based on the image information acquired by the image information acquisition unit 13, an alertness estimation unit 16 that estimates the level of alertness of the driver J1, and an inattentiveness detection unit 19 that detects whether the driver J1 is looking away based on the degree of eyelid opening of the driver J1 detected by the eyelid opening detection unit 14 and the level of alertness of the driver J1 estimated by the alertness estimation unit 16.

[0035] Generally, when detecting an occupant's gaze using image recognition technology based on image information obtained when capturing an image of the occupant's face, if the occupant's face is facing downward rather than forward, it may be more difficult to obtain an image of the occupant's eyes than when the occupant's face is facing forward, depending on the positional relationship between the imaging device and the occupant's face, and the gaze may not be detected. Furthermore, when attempting to detect an occupant's inattentive behavior based on the occupant's gaze, the occupant's inattentive behavior cannot be detected if the occupant's gaze cannot be detected. In contrast, the occupant status detection device 100 according to the first embodiment is configured to detect the driver J1's inattentive behavior based on the degree of eyelid opening and the driver J1's alertness, and therefore can detect the driver J1's inattentive behavior even when the driver J1's gaze cannot be detected. By detecting the driver J1's inattentive behavior, the occupant status detection device 100 according to the first embodiment controls the notification device 120 to issue a message urging the driver J1 not to look away, thereby making it possible to prevent the driver J1 from inattentively driving the vehicle 1.

[0036] Furthermore, as described above, when detecting the degree of eyelid opening using image recognition technology based on image information obtained when capturing an image of the occupant's face, when the occupant's face is facing downward rather than forward and looking aside, the degree of eyelid opening tends to be detected as a lower value than when the occupant's face is facing forward. Therefore, when determining the occupant's state based only on the degree of eyelid opening, it is difficult to determine whether the occupant is facing downward rather than forward and looking aside, or whether the occupant has their eyes closed and is dozing.

[0037] In contrast, the occupant state detection device 100 according to embodiment 1 is configured to detect inattentive driving based on the occupant's level of alertness in addition to the degree to which the occupant's eyelids are open, and therefore, the accuracy of detecting inattentive driving can be improved compared to, for example, detecting inattentive driving based only on the degree to which the occupant's eyelids are open.

[0038] Moreover, the occupant state detection device 100 according to the first embodiment is configured to estimate the alertness of the occupant based on the image information acquired by the image information acquisition unit 13. With this configuration, the occupant state detection device 100 according to the first embodiment does not require a sensor for estimating the alertness and a communication path connecting the sensor and the occupant state detection device, which makes it possible to simplify the device.

[0039] Furthermore, the occupant state detection device 100 according to the first embodiment is configured to detect an occupant looking away when the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit 14 is lower than a preset eyelid opening degree threshold and when the degree of alertness of the occupant estimated by the alertness degree estimation unit 16 is equal to or higher than a preset alertness threshold. With this configuration, the occupant state detection device 100 according to the first embodiment can distinguish between a state in which the occupant is dozing and a state in which the occupant is looking away when, for example, the degree of eyelid opening of the occupant is lower than the threshold, thereby improving the detection accuracy in detecting an occupant looking away compared to conventional devices.

[0040] In the first embodiment, the arousal level estimation unit 16 is configured to estimate the arousal level of the driver J1 based on the image information acquired by the image information acquisition unit 13. However, this is not limiting. The arousal level estimation unit may be configured to estimate the arousal level of any occupant J2 using at least information other than the eyelid opening degree. For example, the arousal level estimation unit may be configured to estimate the arousal level of the occupant J2 based on information acquired by a sensor (not shown) that detects vehicle movement. Specifically, the arousal level estimation unit may be configured to estimate the driver's arousal level by detecting the frequency of sudden deceleration and acceleration of the vehicle based on information from a speed sensor that detects the vehicle speed or information from an acceleration sensor (not shown) that detects the vehicle acceleration. Furthermore, for example, the arousal level estimation unit may be configured to estimate the driver's arousal level by detecting meandering of the vehicle based on information from a sensor (not shown) that detects the operation of the vehicle's steering wheel. Furthermore, for example, the alertness estimation unit may be configured to estimate the driver's alertness by detecting the frequency with which the vehicle crosses a dividing line based on information from a sensor (not shown) that detects dividing lines on the road on which the vehicle is traveling.

[0041] Furthermore, in the first embodiment, the occupant state detection device 100 is configured to detect the occupant J2 looking aside when the eyelid opening degree of the occupant J2 detected by the eyelid opening degree detection unit 14 is lower than a preset threshold and when the wakefulness of the occupant J2 estimated by the wakefulness estimation unit 16 is equal to or higher than the preset threshold, but is not limited to this. The occupant state detection device may be configured to detect the occupant looking aside based on the eyelid opening degree of the occupant detected by the eyelid opening degree detection unit and the wakefulness of the occupant estimated by the wakefulness estimation unit. For example, the occupant state detection device may be configured such that the inattentiveness detection unit detects the occupant looking aside based on an index other than the eyelid opening degree calculated based on the eyelid opening degree of the occupant detected by the eyelid opening degree detection unit and the wakefulness of the occupant estimated by the wakefulness estimation unit. Specifically, the occupant state detection device may be configured such that the inattentiveness detection unit detects the occupant's inattentiveness based on the occupant's eyelid closing time calculated based on the occupant's eyelid opening degree detected by the eyelid opening degree detection unit and the occupant's alertness estimated by the alertness degree estimation unit.

[0042] The occupant state detection device may be configured to calculate the frequency at which the occupant's eyelid opening degree is less than a threshold based on the occupant's eyelid opening degree detected by the eyelid opening degree detection unit, and detect the occupant's looking aside based on a comparison result between the frequency at which the occupant's eyelid opening degree is less than the threshold and a preset frequency threshold, and the occupant's wakefulness estimated by the wakefulness estimation unit. For example, if the occupant's eyelid opening degree is less than the threshold when the wakefulness level is equal to or greater than the threshold, it is possible that the occupant is blinking or looking aside, but if the frequency at which the occupant's eyelid opening degree is less than the threshold is equal to or greater than the threshold, it can be determined that the occupant is frequently looking aside.

[0043] The occupant state detection device may be configured to estimate the occupant's gaze direction based on image information acquired by the image information acquisition unit when the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit is equal to or greater than a threshold, and detect the occupant's inattentive behavior based on the occupant's gaze direction and the occupant's alertness estimated by the alertness estimation unit. For example, if the angle formed by the occupant's gaze direction and the forward direction is equal to or greater than a predetermined threshold and the occupant's alertness is equal to or greater than a threshold, it may be determined that the occupant is looking aside. In this way, various configurations for detecting the occupant's inattentive behavior based on the occupant's eyelid opening degree detected by the eyelid opening degree detection unit and the occupant's alertness estimated by the alertness estimation unit can improve the detection accuracy when detecting the occupant's inattentive behavior compared to conventional methods.

[0044] Furthermore, in the first embodiment, the occupant condition detection device 100 is configured to estimate the wakefulness level of the driver J1, but is not limited thereto. The occupant condition detection device may be configured to detect a state in which at least one occupant J2 is looking away. For example, the occupant condition detection device may be configured to detect looking away by an occupant other than the driver, or may be configured to detect a state other than looking away as an occupant condition. For example, an occupant condition other than looking away may be an occupant falling asleep. Furthermore, for example, by detecting looking away by an occupant other than the driver, it is possible to determine whether the occupant is gazing at a monitor (not shown) located in front of the seat in which the occupant is seated. If the occupant is not gazing at a monitor, the power of the monitor may be turned off or the volume of the monitor may be lowered, thereby reducing the power consumption of the monitor.

[0045] Embodiment 2 Next, an occupant status detection device 200 according to Embodiment 2 will be described with reference to Figures 6 and 7. Figure 6 is a block diagram showing a schematic configuration of the occupant status detection device 200 according to Embodiment 2. The occupant status detection device 200 according to Embodiment 2 differs from the occupant status detection device 100 according to Embodiment 1 in that the inattentive driving detection unit 219 has a configuration for detecting an inattentive driving state of an occupant based on the time period during which the occupant's eyelids are closed. However, the other configurations are the same, and the same configurations as those in Embodiment 1 will be assigned the same names and symbols as those in Embodiment 1, and description thereof will be omitted.

[0046] As shown in Figure 6, the occupant status detection device 200 of embodiment 2 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, and an aside-looking detection unit 219, and is electrically connected to the imaging device 110 and the alarm device 120 so as to be able to communicate with them.

[0047] The eyelid closing time determination unit 15 determines whether the length of the eyelid closing time of the driver J1 is equal to or greater than a preset threshold, based on the eyelid opening degree of the driver J1 detected by the eyelid opening degree detection unit 14. For example, the eyelid closing time determination unit 15 calculates the length of continuous eyelid closing time, which is the time during which the eyelid closing degree of the driver J1 is continuously maintained below a preset threshold, based on the eyelid opening degree of the driver J1 detected by the eyelid opening degree detection unit 14, and determines whether the calculated length of continuous eyelid closing time is equal to or greater than the preset threshold.

[0048] The inattentiveness detection unit 219 detects inattentiveness of the driver J1 based on the determination result by the eyelid closing time determination unit 15 and the wakefulness of the driver J1 estimated by the wakefulness estimation unit 16. For example, the inattentiveness detection unit 219 detects inattentiveness of the driver J1 based on the determination result by the eyelid closing time determination unit 15 and the result of comparing the wakefulness of the driver J1 estimated by the wakefulness estimation unit 16 with a preset wakefulness threshold. Details of the processing performed by the inattentiveness detection unit 219 will be described later.

[0049] 7 is a flowchart showing the processing performed by the occupant state detection device 200 according to Embodiment 2. Note that the processing of steps ST1, ST2, ST3, ST9, and ST13 performed by the occupant state detection device 200 according to Embodiment 2 is similar to the processing performed by the occupant state detection device 100 according to Embodiment 1, and therefore description thereof will be omitted.

[0050] After the process of step ST3, the occupant status detection device 200 next determines whether the eyelid-closed time of the driver J1 is equal to or longer than a preset eyelid-open time threshold (step ST7). In this process, the occupant status detection device 200, for example, compares the continuous eyelid-closed time of the driver J1 with a continuous eyelid-closed time threshold that is considered to indicate that the driver J1 is likely to be awake and facing forward, and determines, using the inattentiveness detection unit 219, whether the length of the continuous eyelid-closed time calculated based on the eyelid opening degree of the driver J1 is long enough to indicate that the driver J1 is not looking aside.

[0051] If the eyelid-closure time of the driver J1 is equal to or greater than a preset threshold in the process of step ST7 (YES in step ST7), the occupant state detection device 200 then determines whether the wakefulness level of the driver J1 is equal to or greater than a preset threshold (step ST9). If the eyelid-closure time of the driver J1 is less than the threshold in the process of step ST7 (NO in step ST7), if the wakefulness level of the driver J1 is less than the threshold in the process of step ST9 (NO in step ST9), or if the process of step ST13 is performed, the occupant state detection device 200 ends the process.

[0052] As described above, the occupant state detection device 200 according to the second embodiment includes the eyelid closing time determination unit 15 that determines whether the length of the continuous eyelid closing time of the occupant is equal to or greater than the preset eyelid closing time threshold, based on the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit 14, and the inattentiveness detection unit 319 is configured to detect the occupant's inattentiveness based on the determination result by the eyelid closing time determination unit 15 and the occupant's alertness estimated by the alertness level estimation unit 16. With this configuration, the occupant state detection device 200 according to the second embodiment can suppress erroneous detection of inattentiveness due to the occupant's blinking and can suppress overdetection, in which a short-term movement of the eyes that does not affect driving is detected as inattentiveness, thereby improving the detection accuracy when detecting inattentiveness of the occupant compared to conventional devices.

[0053] In the second embodiment, the eyelid closing time determination unit 15 is configured to determine whether the length of the continuous eyelid closing time of the driver J1 is equal to or greater than a preset threshold, based on the eyelid opening degree of the driver J1 detected by the eyelid opening degree detection unit 14. However, this is not limiting. The eyelid closing time determination unit may be configured to determine whether the length of the eyelid closing time of the occupant is equal to or greater than a preset threshold, based on the eyelid opening degree of the driver J1 detected by the eyelid opening degree detection unit. For example, the eyelid closing time determination unit may be configured to calculate the total time during a specific period during which the eyelid closing degree of the occupant is less than the preset threshold, based on the eyelid opening degree of the occupant detected by the eyelid opening degree detection unit, and determine whether the proportion of the total time during the specific period is equal to or greater than a preset threshold. Furthermore, for example, the eyelid closing time determination unit may be configured to ignore opening and closing of the eyelids for a period of time less than a predetermined threshold when calculating the length of time the occupant's eyelids are continuously closed based on the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit.

[0054] Embodiment 3 Next, an occupant status detection device 300 according to Embodiment 3 will be described with reference to Figures 8 and 9. Figure 8 is a block diagram showing a schematic configuration of the occupant status detection device 300 according to Embodiment 3. The occupant status detection device 300 according to Embodiment 3 differs from the occupant status detection device 200 according to Embodiment 2 in the configuration in which the inattentive driving detection unit 319 detects an occupant's inattentive driving based on the direction of the occupant's face, but the other configurations are the same, and the same configurations as those in Embodiment 2 will be assigned the same names and symbols as those in Embodiment 2 and will not be described again.

[0055] As shown in Figure 8, the occupant status detection device 300 of embodiment 3 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, a face direction estimation unit 17, and an aside looking detection unit 319, and is electrically connected so as to be able to communicate with the imaging device 110 and the alarm device 120.

[0056] The face direction estimation unit 17 estimates the face direction of the driver J1 based on the image information acquired by the image information acquisition unit 13. For example, the face direction estimation unit 17 estimates the angle formed by the face direction of the driver J1 and the forward direction based on the image information acquired by the image information acquisition unit 13 and a trained model that learns using training data that associates image information obtained by capturing an image of at least the face of the occupant with the face direction of the occupant, and outputs the angle of the face direction of the occupant relative to the forward direction in response to input image information including an image of the occupant's face. Furthermore, for example, the face direction estimation unit 17 estimates the face direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a known image recognition technology. Specifically, the face direction estimation unit 17 extracts facial feature points of the driver J1 present in a detection area of ​​the image information acquired by the image information acquisition unit 13, where the face of the driver J1 may be present, by using a known edge detection technology in a state where the detection area is set in advance. The detection area is an area in which the face of the driver J1 may be present in the image information. Next, the face direction estimation unit 17 estimates the face direction of the driver J1 based on the extracted feature points of each part of the face of the driver J1.

[0057] The inattentiveness detection unit 319 detects the inattentiveness of the driver J1 based on the determination result by the eyelid closing time determination unit 15, the alertness of the driver J1 estimated by the alertness estimation unit 16, and the facial direction of the driver J1 estimated by the facial direction estimation unit 17. For example, the inattentiveness detection unit 219 detects the inattentiveness of the driver J1 based on the determination result by the eyelid closing time determination unit 15, the result of comparing the alertness of the driver J1 estimated by the alertness estimation unit 16 with a preset threshold value of the alertness, and the determination result of whether the facial direction of the driver J1 estimated by the facial direction estimation unit 17 is within a preset range. Details of the processing performed by the inattentiveness detection unit 319 will be described later.

[0058] 9 is a flowchart showing the processing performed by the occupant state detection device 300 according to Embodiment 3. Note that the processing of steps ST1, ST2, ST3, ST7, ST9, and ST13 performed by the occupant state detection device 300 according to Embodiment 3 is similar to the processing performed by the occupant state detection device 200 according to Embodiment 2, and therefore description thereof will be omitted.

[0059] After the processing of step ST3, the occupant state detection device 300 next estimates the facial direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 (step ST4). In this processing, the occupant state detection device 300 estimates the facial direction of the driver J1 by the facial direction estimation unit 17 based on the image information acquired by the image information acquisition unit 13.

[0060] After the process of step ST4, the occupant status detection device 300 next determines whether the eyelid-closed time of the driver J1 is equal to or greater than a preset threshold for the eyelid-open time (step ST7). If the eyelid-closed time of the driver J1 is equal to or greater than the preset threshold in the process of step ST7 (YES in step ST7), the occupant status detection device 300 next determines whether the wakefulness level of the driver J1 is equal to or greater than a preset threshold (step ST9). If the wakefulness level of the driver J1 is equal to or greater than the threshold in the process of step ST9 (YES in step ST9), the occupant status detection device 300 next detects that the driver J1 is looking away (step ST32).

[0061] If the wakefulness level of the driver J1 is below the threshold in the process of step ST9 (NO in step ST9), the occupant state detection device 300 then determines whether the facial direction of the driver J1 estimated by the facial direction estimation unit 17 is tilted downward by more than a predetermined range (step ST10). For example, in this process, the occupant state detection device 300 compares the angle between the facial direction of the driver J1 estimated by the facial direction estimation unit 17 and the front with a predetermined angle threshold, and determines whether the angle between the facial direction of the driver J1 and the front is equal to or greater than the threshold using the inattentive driving detection unit 319. For example, the angle threshold is set as the angle between the facial direction of a driver who is considered to be highly likely to be not looking aside and be awake and the front. Furthermore, for example, the angle threshold is set as the angle between the facial direction of the occupant and the front, which is detected by the eyelid opening degree detection unit 14 as an eyelid opening degree equal to or greater than the threshold when the eyelid opening degree of the occupant is not looking aside and is awake.

[0062] In the process of step ST10, if the driver J1's facial orientation is not more downward than a predetermined range (NO in step ST10), the occupant status detection device 300 then detects that the driver J1 is looking aside (step ST32). In this process, the occupant status detection device 300 determines that the driver J1 is not at least looking forward with his / her eyelids open, based on the fact that the eyelid closing time of the driver J1 is more than a threshold, the driver J1's alertness level is less than a threshold, and the driver J1's facial orientation is not more downward than a predetermined range, and therefore determines that the driver J1 is looking aside using the inattentiveness detection unit 319. After the process of step ST32 is performed, the occupant status detection device 300 then outputs the detection result of the driver J1's looking aside to the alarm device 120 (step ST13).

[0063] In the processing of step ST7, if the eyelid closure time of driver J1 is less than the threshold value (NO in step ST7), if the facial direction of driver J1 is more downward than the preset range in the processing of step ST10 (YES in step ST10), or if the processing of step ST13 is performed, the occupant status detection device 300 terminates the processing.

[0064] As described above, the occupant state detection device 300 according to the third embodiment includes the face direction estimation unit 17 that estimates the facial direction of the occupant based on the image information acquired by the image information acquisition unit 13, and the inattentiveness detection unit 319 is configured to detect the inattentiveness of the occupant based on the facial direction of the occupant estimated by the face direction estimation unit 17. As configured in this way, the occupant state detection device 300 according to the third embodiment is configured to detect the inattentiveness of the occupant based on the facial direction of the occupant in addition to the degree of eyelid opening and the alertness of the occupant, and therefore can improve the detection accuracy when detecting the inattentiveness of the occupant compared to when the inattentiveness of the occupant is detected based only on the degree of eyelid opening and the alertness of the occupant.

[0065] In the third embodiment, the occupant state detection device 300 is configured to determine that the driver J1 is looking aside when the facial direction of the driver J1 estimated by the facial direction estimation unit 17 is not downward by more than a predetermined range, but is not limited to this. The occupant state detection device may be configured to detect the occupant's looking aside based on the degree of eyelid opening, the occupant's alertness, and the occupant's facial direction, and may be configured to determine that the occupant is looking aside when, for example, the occupant's eyelids are closed for less than a threshold, the occupant's alertness is equal to or greater than a threshold, and the angle between the occupant's facial direction and the front is equal to or greater than a predetermined threshold.

[0066] Fourth Embodiment Next, an occupant status detection device 400 according to a fourth embodiment will be described with reference to Figures 10 and 11. Figure 10 is a block diagram showing a schematic configuration of the occupant status detection device 400 according to the fourth embodiment. The occupant status detection device 400 according to the fourth embodiment differs from the occupant status detection device 300 according to the third embodiment in that it includes a drowsiness detection unit 20 for detecting drowsiness by an occupant, but the other configurations are similar. Therefore, the same names and symbols as those in the third embodiment will be used and descriptions thereof will be omitted.

[0067] As shown in Figure 10, the occupant status detection device 400 of embodiment 4 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, a facial direction estimation unit 17, an aside-looking detection unit 319, and a drowsiness detection unit 20, and is electrically connected to the imaging device 110 and the alarm device 120 so as to be able to communicate with them.

[0068] The drowsiness detection unit 20 detects that the driver J1 is dozing off based on the eyelid opening degree of the driver J1 detected by the eyelid opening degree detection unit 14, the wakefulness of the driver J1 estimated by the wakefulness estimation unit 16, and the facial direction of the driver J1 estimated by the facial direction estimation unit 17. For example, the drowsiness detection unit 20 detects that the driver J1 is dozing off when the wakefulness of the driver J1 estimated by the wakefulness estimation unit 16 is lower than a preset wakefulness threshold. Details of the processing performed by the drowsiness detection unit 20 will be described later.

[0069] 11 is a flowchart showing the processing performed by the occupant status detection device 400 according to Embodiment 4. Note that the processing of steps ST1, ST2, ST3, ST4, ST7, ST9, ST10, ST32, and ST13 performed by the occupant status detection device 400 according to Embodiment 4 is similar to the processing performed by the occupant status detection device 300 according to Embodiment 3, and therefore description thereof will be omitted.

[0070] In the process of step ST10, if the driver J1's face is facing downward by more than a predetermined range (YES in step ST10), the occupant state detection device 400 detects that the driver J1 is dozing off (step ST11). In this process, the occupant state detection device 400 determines that the driver J1 is dozing off by the dozing detection unit 20 based on the fact that the eyelid closing time of the driver J1 is equal to or longer than a threshold, the wakefulness level of the driver J1 is less than a threshold, and the face direction of the driver J1 is facing downward by more than a predetermined range.

[0071] If the process of step ST32 has been performed, or if the process of step ST11 has been performed, then the occupant state detection device 400 outputs the detection result of the driver J1 looking aside and the detection result of the driver J1 falling asleep to the alarm device 120 (step ST13). If the eyelid closure time of the driver J1 is less than the threshold value in the process of step ST7 (NO in step ST7), or if the process of step ST13 has been performed, the occupant state detection device 400 ends the process.

[0072] As described above, the occupant state detection device 400 according to the fourth embodiment includes the drowsiness detection unit 20 that detects whether the occupant is dozing off, based on the degree of eyelid opening of the occupant detected by the eyelid opening detection unit 14, the level of alertness of the occupant estimated by the alertness estimation unit 16, and the facial direction of the occupant estimated by the facial direction estimation unit 17. With this configuration, the occupant state detection device 400 according to the fourth embodiment is able to distinguish between the occupant looking away and the occupant dozing, thereby enabling the notification device 120 to notify the occupant of information according to the state of the occupant.

[0073] In the fourth embodiment, the occupant state detection device 400 is configured to determine, by the drowsiness detection unit 20, that the driver J1 is dozing off based on the fact that the eyelid closing time of the driver J1 is equal to or longer than a threshold, the alertness level of the driver J1 is less than a threshold, and the face direction of the driver J1 is more downward than a preset range, but is not limited to this. The occupant state detection device may be configured to detect the occupant's dozing off based on the eyelid opening degree of the occupant detected by the eyelid opening degree detection unit, the alertness level of the occupant estimated by the alertness level estimation unit, and the face direction of the occupant estimated by the face direction estimation unit. For example, the occupant state detection device may be configured to determine that the occupant is dozing off based on the fact that the eyelid opening degree of the occupant is less than a threshold, the alertness level of the occupant is less than a threshold, and the face direction of the occupant is not more downward than a preset range.

[0074] Embodiment 5 Next, an occupant status detection device 500 according to Embodiment 5 will be described with reference to Figures 12 and 13. Figure 12 is a block diagram showing a schematic configuration of the occupant status detection device 500 according to Embodiment 5. The occupant status detection device 500 according to Embodiment 5 differs from the occupant status detection device 400 according to Embodiment 3 in that the inattentive driving detection unit 519 has a different configuration for detecting an inattentive driving state of an occupant based on the line of sight of the occupant. However, the other configurations are the same, and the same configurations as those in Embodiment 4 are assigned the same names and symbols as those in Embodiment 4, and description thereof will be omitted.

[0075] As shown in Figure 12, the occupant status detection device 500 of embodiment 5 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, a facial direction estimation unit 17, a gaze detection unit 18, an aside-looking detection unit 519, and a drowsiness detection unit 20, and is electrically connected to the imaging device 110 and the alarm device 120 so as to be able to communicate with them.

[0076] The gaze detection unit 18 detects the gaze direction of the driver J1 by estimating the facial orientation of the driver J1 based on the image information acquired by the image information acquisition unit 13. For example, the gaze detection unit 18 estimates the gaze direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a trained model that learns using training data that associates image information obtained by capturing an image of at least the face of the occupant with the gaze direction of the occupant, and outputs the gaze direction of the occupant in response to input image information including an image of the occupant's face. Also, for example, the gaze detection unit 18 estimates the gaze direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 and known image recognition technology. Specifically, the gaze detection unit 18 extracts feature points indicating the upper eyelids, the lower eyelids, and the pupils of the driver J1 that are present in the detection area of ​​the image information acquired by the image information acquisition unit 13, by using a known edge detection technique in a state where a detection area is previously set as an area where the face of the driver J1 may be present in the image information acquired by the image information acquisition unit 13. Next, the face direction estimation unit 17 estimates the gaze direction of the driver J1 based on the positional relationship between the extracted left and right edges of the upper and lower eyelids of the driver J1 and the center of the pupil.

[0077] The inattentiveness detection unit 519 detects inattentiveness of the driver J1 based on the determination result by the eyelid closing time determination unit 15, the alertness of the driver J1 estimated by the alertness level estimation unit 16, the facial direction of the driver J1 estimated by the facial direction estimation unit 17, and the gaze direction of the driver J1 detected by the gaze detection unit 18. Details of the processing performed by the inattentiveness detection unit 519 will be described later.

[0078] 13 is a flowchart showing the processing performed by the occupant status detection device 500 according to Embodiment 5. Note that the processing of steps ST1, ST2, ST3, ST4, ST7, ST9, ST10, ST11, and ST13 performed by the occupant status detection device 500 according to Embodiment 5 is similar to the processing performed by the occupant status detection device 400 according to Embodiment 4, and therefore description thereof will be omitted.

[0079] After the process of step ST4, the occupant status detection device 500 then detects the gaze direction of the driver J1 (step ST5). In this process, the occupant status detection device 500 detects the gaze direction of the driver J1 by calculating the gaze direction of the driver J1 using the gaze detection unit 18 based on the image information acquired by the image information acquisition unit 13.

[0080] After the processing of step ST5, the occupant status detection device 500 next determines whether the angle formed between the line of sight of the driver J1 and the road ahead is less than a preset threshold angle (step ST8). In this processing, for example, the occupant status detection device 500 determines whether the driver J1 is looking aside by using the inattentiveness detection unit 519, by comparing the line of sight of the driver J1, whose eyelid opening degree detected by the line of sight detection unit 18 is equal to or greater than a threshold angle, with a threshold angle which is the angle formed between the line of sight and the road ahead at which the driver J1 is considered not to be looking aside.

[0081] If the angle between the driver J1's gaze direction and the road ahead is equal to or greater than a predetermined threshold angle in the process of step ST8 (NO in step ST8), the occupant status detection device 500 then detects that the driver J1 is looking aside (step ST52). In this process, the occupant status detection device 500 determines, using the inattentiveness detection unit 519, that the driver J1 is looking aside based on the fact that the angle between the driver J1's gaze direction and the road ahead is equal to or greater than a predetermined threshold angle when the eyelid closure time is less than the threshold. For example, if an attempt is made to detect the occupant's gaze direction based on image information when the occupant's eyelid closure time is equal to or greater than the threshold, sufficient detection accuracy may not be achieved. Therefore, even if a calculation result of the occupant's gaze direction is obtained based on image information, the calculated gaze direction is likely to differ significantly from the direction the occupant is actually looking. The occupant status detection device 500 according to the fifth embodiment detects that the driver J1 is looking away based on whether the eyelids of the driver J1 are closed for a period of time that is less than a threshold value and whether the angle between the line of sight of the driver J1 and the front is equal to or greater than a preset threshold angle, thereby improving the accuracy of detecting inattentive behavior compared to conventional methods.

[0082] If the process of step ST52 has been performed, or if the process of step ST11 has been performed, then the occupant status detection device 500 outputs the detection result of the driver J1 looking aside and the detection result of the driver J1 falling asleep to the alarm device 120 (step ST13). If the angle formed between the line of sight of the driver J1 and the front is less than the threshold value in the process of step ST8 (YES in step ST8), or if the process of step ST13 has been performed, the occupant status detection device 500 ends the process.

[0083] As described above, the occupant state detection device 500 according to the fifth embodiment is configured to detect an occupant looking aside based on the determination result by the eyelid closing duration determination unit 15, the occupant's alertness estimated by the alertness estimation unit 16, the occupant's facial direction estimated by the facial direction estimation unit 17, and the occupant's gaze direction detected by the gaze detection unit 18. With this configuration, the occupant state detection device 500 according to the fifth embodiment is configured to detect an occupant looking aside based on the gaze direction when sufficient detection accuracy of the gaze direction can be obtained, and therefore can improve the detection accuracy of inattentive behavior compared to conventional devices.

[0084] Note that the occupant state detection device 500 according to the fifth embodiment is configured to detect the occupant's inattentive behavior based on the determination result by the eyelid closure duration determination unit 15, the occupant's alertness estimated by the alertness estimation unit 16, the occupant's facial direction estimated by the facial direction estimation unit 17, and the occupant's gaze direction detected by the gaze detection unit 18, but is not limited to this. The occupant state detection device may be configured to detect the occupant's inattentive behavior based on the occupant's eyelid closure degree, the occupant's alertness, the occupant's facial direction, and the occupant's gaze direction. For example, the occupant state detection device may be configured to detect the occupant's inattentive behavior based on the gaze direction when the occupant's eyelid closure degree is equal to or greater than a threshold.

[0085] In any of the above-described embodiments, the occupant status detection device may have some or all of the functions of the imaging device and the alarm device, or the imaging device and the alarm device may have some of the functions of the occupant status detection device.

[0086] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0087] The occupant status detection device according to the present disclosure can be used, for example, in a system that detects when an occupant in a vehicle is looking away and issues an alert to warn the occupant not to look away.

[0088] 1 Vehicle, 2 Driver's seat, 3 Seat, 11 Input unit, 12 Output unit, 13 Image information acquisition unit, 14 Eyelid opening detection unit, 15 Eyelid closing time determination unit, 16 Arousal level estimation unit, 17 Face direction estimation unit, 18 Gaze detection unit, 19 Inattention detection unit, 20 Detection unit, 100 Occupant state detection device, 110 Imaging device, 120 Alarm device, 200 Occupant state detection device, 219 Inattention detection unit, 300 Occupant state detection device, 319 Inattention detection unit, 400 Occupant state detection device, 500 Occupant state detection device, 519 Inattention detection unit, J1 Driver, J2 Occupant, R Vehicle compartment.

Claims

1. An image information acquisition unit that acquires image information obtained by imaging the interior of the vehicle, An eyelid opening degree detection unit detects the degree of eyelid opening of an occupant inside the vehicle based on the image information acquired by the image information acquisition unit, A unit for estimating the level of alertness of the aforementioned crew member, The system includes a distraction detection unit that detects the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit and the occupant's level of alertness estimated by the alertness estimation unit, The distraction detection unit detects the occupant's distraction when the degree of eyelid opening detected by the eyelid opening detection unit is lower than a preset threshold for eyelid opening, and the occupant's level of alertness estimated by the alertness estimation unit is equal to or greater than a preset threshold for alertness. A passenger status detection device characterized by the following features.

2. The alertness estimation unit estimates the alertness level of the crew member based on the image information acquired by the image information acquisition unit. The occupant status detection device according to feature 1.

3. The system includes a closed-eye time determination unit that determines whether the length of the occupant's closed-eye time is equal to or greater than a preset closed-eye time threshold, based on the degree of eyelid opening detected by the eyelid opening degree detection unit. The distraction detection unit detects the occupant's distraction based on the determination result by the eye-closing time determination unit. The occupant status detection device according to feature 1.

4. The distraction detection unit detects the occupant's distraction if the determination result from the eye-closing time determination unit indicates that the occupant's eye-closing time is longer than a preset time, and the occupant's level of alertness estimated by the alertness estimation unit is higher than a preset alertness threshold. The occupant status detection device according to feature 3.

5. The system includes a face orientation estimation unit that estimates the face orientation of the occupant based on the image information acquired by the image information acquisition unit, The distraction detection unit detects the occupant's distraction based on the occupant's facial orientation estimated by the facial orientation estimation unit. The occupant status detection device according to any one of claims 1 to 4.

6. The system includes a drowsiness detection unit that detects the occupant's drowsiness based on the degree of eyelid opening detected by the eyelid opening detection unit, the occupant's level of alertness estimated by the alertness estimation unit, and the occupant's facial orientation estimated by the facial orientation estimation unit. The occupant status detection device according to claim 5, characterized in that it is a feature of the present invention.

7. The drowsiness detection unit detects drowsiness in the occupant if the occupant's level of alertness, as estimated by the alertness estimation unit, is lower than a preset threshold for alertness. The occupant status detection device according to feature 6.

8. The distraction detection unit detects the occupant's distraction when the occupant's level of alertness estimated by the alertness estimation unit is higher than a preset alertness threshold, and the occupant's facial orientation estimated by the facial orientation estimation unit is outside a preset range. The occupant status detection device according to feature 6.

9. Computers, An image information acquisition unit that acquires image information obtained by imaging the interior of the vehicle, An eyelid opening degree detection unit detects the degree of eyelid opening of an occupant inside the vehicle based on the image information acquired by the image information acquisition unit, A unit for estimating the level of alertness of the aforementioned crew member, A program that causes a distraction detection unit to function as a unit that detects the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit and the occupant's level of alertness estimated by the alertness level estimation unit, The distraction detection unit detects the occupant's distraction when the degree of eyelid opening detected by the eyelid opening detection unit is lower than a preset threshold for eyelid opening, and the occupant's level of alertness estimated by the alertness estimation unit is equal to or greater than a preset threshold for alertness. A crew status detection program characterized by the following features.

10. A method for detecting the state of an occupant, performed by a device comprising an image information acquisition unit, an eyelid opening degree detection unit, an alertness level estimation unit, and a distracted-looking detection unit, The image information acquisition unit performs the step of acquiring image information obtained by imaging the interior of the vehicle, The eyelid opening degree detection unit detects the degree of eyelid opening of the occupant inside the vehicle based on the image information acquired by the image information acquisition unit, The alertness estimation unit performs the steps of estimating the alertness level of the crew member, The distraction detection unit includes the step of detecting the occupant's distraction based on the degree of eyelid opening detected by the eyelid opening detection unit and the occupant's level of alertness estimated by the alertness estimation unit, The distraction detection unit detects the occupant's distraction when the degree of eyelid opening detected by the eyelid opening detection unit is lower than a preset threshold for eyelid opening, and the occupant's level of alertness estimated by the alertness estimation unit is equal to or greater than a preset threshold for alertness. A method for detecting the condition of an occupant, characterized by the features described above.