Occupant condition detection device and occupant condition detection method
The occupant condition detection device accurately assesses inattentiveness by accounting for temporary factors affecting line of sight detection, improving the reliability of inattentiveness detection in vehicle occupants.
Patent Information
- Application Number
- JP2023575342
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-01-27
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-01-27
AI Technical Summary
Existing occupant condition detection systems inaccurately determine inattentiveness when the driver's line of sight is directed away from the road due to temporary factors like blinking or low light conditions, despite the driver's face being oriented forward.
An occupant condition detection device that utilizes an imaging device to capture vehicle occupant images, detects line of sight and facial orientation, estimates factors impeding detection (like blinking or low light), and determines inattentiveness based on both current and past frames to prevent false negatives.
Prevents immediate false determinations of inattentiveness by considering temporary factors, enhancing the accuracy of inattentiveness detection and ensuring timely warnings when necessary.
Smart Images

Figure 0007594843000001 
Figure 0007594843000002 
Figure 0007594843000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an occupant state detection technology. [Background technology]
[0002] Patent Document 1 discloses a visually-identified object detection device that includes a camera that photographs a driver, a face direction detection unit that detects the face direction of the driver from an image photographed by the camera, a gaze detection unit that detects the driver's gaze from the image photographed by the camera, a gaze estimation unit that estimates the driver's gaze from the face direction, a gaze target estimation unit that estimates a gaze target that the driver is viewing from the gaze, and an output unit that outputs information indicating the gaze target estimated by the gaze target estimation unit, wherein the gaze target estimation unit estimates the gaze target based on the gaze detected by the gaze detection unit when the detection accuracy of the gaze detected by the gaze detection unit is equal to or higher than a predetermined threshold, and estimates the gaze from the face direction detected by the face direction detection unit and estimates the gaze target based on the gaze estimated by the gaze estimation unit when the detection accuracy of the gaze is lower than the predetermined threshold. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2019-101520 A Summary of the Invention [Problem to be solved by the invention]
[0004] Assume that the driver's face is facing forward of the vehicle, but the driver's gaze is directed to the side. In such a case, according to the device of the above-mentioned Patent Document 1, when the gaze detection accuracy is equal to or higher than a predetermined threshold and the gaze can be acquired, the gaze is directed to the side, so it is determined that the driver is inattentive due to looking away. In such a state where the driver is determined to be inattentive, if the driver's gaze cannot be acquired due to some factor such as blinking, the device of Patent Document 1 executes a determination based on the face direction. Therefore, there is a problem in that, even if the driver is actually inattentive due to looking away, the determination that the driver is inattentive is cancelled and the driver is determined to be not inattentive.
[0005] The present disclosure has been made to solve such problems, and aims to provide an occupant state detection technology that can prevent an occupant of a vehicle from immediately determining that they are not inattentive based on the direction of their face when their gaze cannot be detected when they are inattentive. [Means for solving the problem]
[0006] An occupant status detection device according to an embodiment of the present disclosure includes an image acquisition unit configured to acquire a plurality of frames of captured images acquired by an imaging device arranged in a vehicle; a feature information detection unit configured to detect the line of sight and facial direction of an occupant of the vehicle from the captured images of the plurality of frames; a factor estimation unit configured to estimate factors that prevent detection of the line of sight of the occupant for the captured images of the plurality of frames and output information indicating the estimated factors as factor information; and an inattention state determination unit configured to determine whether or not the occupant is in an inattentive state that is unsuitable for driving the vehicle based on the detected line of sight and facial direction and all or a part of the output factor information for the captured images of each frame, and the inattention state determination unit is configured to determine whether or not the occupant is in an inattentive state for the captured image of a current frame based on the factor information output for the captured image of the current frame, the facial direction detected from the captured image of the current frame, and a determination result of an inattentive state made for an captured image of a past frame prior to the current frame. Effect of the Invention
[0007] According to an occupant status detection device according to an embodiment of the present disclosure, when an occupant of a vehicle is inattentive and the line of sight cannot be detected, it is possible to prevent an immediate determination that the occupant is not inattentive based on the direction of the face. [Brief description of the drawings]
[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of an occupant state detection device and an occupant state detection system; [Figure 2A] FIG. 2 is a diagram illustrating an example of a hardware configuration of an occupant state detection device. [Figure 2B] FIG. 2 is a diagram illustrating an example of a hardware configuration of an occupant state detection device. [Diagram 3] 4 is a flowchart showing the operation of the occupant status detection device. [Figure 4] 4 is a flowchart showing the operation of the occupant status detection device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that components with the same or similar reference numerals in the drawings have the same or similar configurations or functions, and redundant descriptions of such components will be omitted.
[0010] Embodiment 1 <Configuration of occupant condition detection system> An occupant state detection device and an occupant state detection system according to the present disclosure will be described with reference to FIG. 1. As shown in FIG. 1, the occupant state detection system includes an occupant state detection device 1 and an imaging device 2. The imaging device 2 is disposed at a position where it can capture an image of the interior of the vehicle, and captures an image of an occupant of the vehicle, typically the driver. For example, an infrared camera can be used as the imaging device 2 when infrared light is irradiated onto the occupant to obtain a corneal reflex. An RGB camera may be used in a bright environment. The imaging device 2 outputs the captured image to the occupant state detection device 1.
[0011] <Configuration of occupant condition detection device> As shown in FIG. 1, the occupant state detection device 1 includes an image acquisition unit 11, a feature information detection unit 12, a cause estimation unit 13, and an inattention state determination unit 14, and determines whether or not the occupant is in an inattentive state based on an image captured by an imaging device 2.
[0012] (Image acquisition section) The image acquisition unit 11 acquires a plurality of time-sequential frames of captured images of a vehicle occupant from the imaging device 2. The image acquisition unit 11 outputs the acquired captured images.
[0013] (Feature information detection section) The feature information detection unit 12 detects at least one of the gaze points and the facial direction of the occupant from the captured image output from the image acquisition unit 11. As an example, the feature information detection unit 12 detects the gaze point and the facial direction by calculating the shape of the facial parts related to the occupant's face, such as the face, eyes, and pupils, and the amount of change over time of the facial parts from the captured image. The gaze point can be calculated, for example, by using the position of the pupil relative to the positions of the inner corner and outer corner of the eye. In addition, the corneal reflex may be detected and the position of the corneal reflex may be used in the calculation, and the corneal reflex can be detected by extracting a high-luminance area within a low-luminance area. The gaze point can be specified by the angle of the gaze point with respect to a virtual straight line connecting the imaging device 2 and the center of either the left or right eye of the occupant. The facial direction can be calculated from the relative positional relationship of the facial parts.
[0014] As another example, the feature information detection unit 12 may detect the gaze and facial direction by inputting the captured image output from the image acquisition unit 11 into a trained model (not shown) that has learned the relationship between the captured image and the gaze and facial direction.
[0015] The feature information detection unit 12 outputs feature information indicating the detected feature points, gaze information indicating the detected gaze, and face direction information indicating the detected face direction.
[0016] (Factor Estimation Section) The factor estimating unit 13 is a functional unit that estimates a factor that hinders gaze detection by the feature information detecting unit 12, and outputs the estimated factor, which is an estimation result, as factor information.
[0017] As an example, the factor estimation unit 13 estimates whether the factor that prevents the gaze detection is blinking, and outputs the estimation result as factor information. When the occupant temporarily closes their eyes by blinking, the gaze cannot be detected, so it estimates whether the occupant is blinking. To perform such an estimation, the factor estimation unit 13 calculates the degree of eye opening of the occupant using the feature information output from the feature information detection unit 12, and judges whether the occupant is blinking. The degree of eye opening can be calculated, for example, by detecting the position of the inner corner of the eye, the position of the outer corner of the eye, and the position of the highest point of the upper eyelid, and dividing the distance between the straight line connecting the inner corner and the outer corner of the eye and the highest point of the upper eyelid by the distance between the inner corner and the outer corner of the eye to obtain the flattening of the eye, and dividing the flattening by a predetermined reference value. The highest point of the upper eyelid is the point (vertex) of the upper eyelid that is farthest from the straight line connecting the outer corner and the inner corner of the eye. When the degree of eye opening falls below a predetermined threshold TH1, it is determined that the eye is closed, when the degree of eye opening exceeds a predetermined threshold TH2 that is greater than the predetermined threshold TH1, it is determined that the eye is open, and when an eye is opened within a predetermined time (hereinafter, may be referred to as "eye closing time") after detecting eye closing, it is determined that "blinking has occurred." When an eye is not opened within the eye closing time after detecting eye closing, it is determined that "blinking has not occurred." When determining that a blink has occurred, the factor estimation unit 13 turns the blink flag ON, and when determining that a blink has not occurred, it turns the blink flag OFF and outputs the blink flag as factor information.
[0018] In this way, when estimating the presence or absence of a blink, the factor estimating section 13 estimates the presence or absence of a blink from a plurality of consecutive frames of captured images corresponding to the eye-closure time.
[0019] As another example, the factor estimation unit 13 estimates whether the factor that prevents the gaze detection is dozing, and outputs the estimation result as factor information. When the occupant has his / her eyes closed due to dozing, the gaze cannot be detected, so it estimates whether the occupant is dozing. To perform such an estimation, the factor estimation unit 13 calculates the degree of eye opening of the occupant using the feature information output from the feature information detection unit 12, and determines whether the occupant is dozing. After detecting the closed eyes, the factor estimation unit 13 can determine that the occupant is dozing if the closed eyes continue beyond the eye closing time used in the blink determination. A time longer than the eye closing time used in the blink determination may be used as the standard. When the factor estimation unit 13 determines that the occupant is dozing, it sets a dozing flag ON, and when it determines that the occupant is not dozing, it sets the dozing flag OFF, and outputs the dozing flag as factor information. In this disclosure, the term "dozing" is used to include not only the case where the occupant is actually dozing, but also a state of drowsiness that is a precursor to dozing.
[0020] As another example, the factor estimation unit 13 estimates whether the factor that prevents the gaze detection is an abnormality in the luminance of the image, and outputs the estimation result as the factor information. For example, when the luminance of the eye area in the captured image is low, the pupil and its surroundings cannot be distinguished and the pupil cannot be detected, so that the gaze may not be detected. Also, when the luminance of the eye area in the captured image is high, the corneal reflection cannot be acquired and the gaze may not be detected. Therefore, the factor estimation unit 13 estimates whether the luminance is abnormal. To perform such an estimation, the factor estimation unit 13 determines that the luminance is abnormal when the luminance value of the captured image is equal to or less than a predetermined first luminance value or equal to or more than a predetermined second luminance value that is greater than the first luminance value. When the luminance value of the captured image is greater than the first luminance value and less than the second luminance value, the factor estimation unit 13 determines that the luminance is normal. As the luminance value of the captured image, the luminance value of the eye area of the occupant in the captured image is typically used, but the luminance value of an area larger than the eye area, for example, the luminance value of the area of the eyes and eyelids, or the luminance value of the area of the eyes, eyelids, and eyebrows may be used. The factor estimation unit 13 turns a brightness flag ON when it determines that the brightness of the image is abnormal, and turns the brightness flag OFF when it determines that the brightness of the image is normal, and outputs the brightness flag as factor information.
[0021] As another example, the factor estimation unit 13 estimates whether the factor that prevents the gaze detection is a glasses reflection, and outputs the estimation result as factor information. For example, when the occupant wears glasses or sunglasses, a reflection called glasses reflection occurs on the glasses or sunglasses due to sunlight or light reflected by various structures such as buildings around the vehicle, and the corneal reflection is hidden by the glasses reflection, so that the gaze cannot be detected. Therefore, the factor estimation unit 13 estimates whether the gaze is a glasses reflection. To perform such an estimation, for example, the factor estimation unit 13 is configured to acquire the gazes of both eyes of the occupant from the characteristic information detection unit 12, and estimates that there is a glasses reflection when only the gaze of one eye of the occupant can be acquired from the characteristic information detection unit 12. Since the occupant is generally seated on either the left or right door of the vehicle, glasses reflection often occurs only on one of the glasses of the glasses or sunglasses. Therefore, when only the gaze of one eye of the occupant can be acquired, it can be estimated that glasses reflection has occurred on the glass in front of the other eye. Glasses reflection may be estimated by other methods such as detecting the shape of a reflecting object reflected in the glasses. The factor estimation unit 13 turns on an eyeglasses reflection flag when it determines that eyeglasses reflection is present, and turns off the eyeglasses reflection flag when it determines that eyeglasses reflection is not present, and outputs the eyeglasses reflection flag as factor information.
[0022] In the above, we have explained four examples of factors that may hinder gaze detection: blinking, dozing, abnormal brightness, and glasses reflection. However, factors other than these may also be estimated and processed in the same manner as in the case of blinking or other factors.
[0023] (Inattention state determination unit) The inattention state determination unit 14 determines whether or not the occupant is in an inattentive state based on all or a part of the gaze information and face direction information output from the feature information detection unit 12 and the factor information output from the factor estimation unit 13. To make such a determination, the inattention state determination unit 14 performs the following determinations: 1) when gaze information is received, determine whether or not the gaze indicated by the gaze information is within a driving area that is an area suitable for driving; 2) when face direction information is received, determine whether or not the face direction indicated by the face direction information is within a driving area; and 3) when factor information is received, determine whether or not the occupant is in an inattentive state based on the factor information. Note that the driving area when gaze information is used may be different from the driving area when face direction information is used.
[0024] To perform judgment 1, the inattention state judgment unit 14 turns on a gaze flag when the gaze indicated by the gaze information is not within the driving area, and turns off the gaze flag when the gaze is within the driving area.
[0025] To perform judgment 2, the inattention state judgment unit 14 turns on a face direction flag, for example, when the face direction indicated by the face direction information is not within the driving area, and turns off the face direction flag when it is within the driving area.
[0026] In order to perform the judgment 3, for example, when either the blink flag or the brightness flag is ON, and the face direction flag is OFF, the inattention state judgment unit 14 judges whether the occupant in the current frame is in an inattention state based on whether the occupant has been judged to be in an inattention state within the past few frames. When the gaze of an occupant who is inattentive driving and whose face direction is within the driving area but whose gaze is not within the driving area blinks, according to the conventional technology, the inattention judgment is performed based on the face direction of the occupant, and if the face direction of the occupant is within the driving area, it is immediately judged that the occupant is not in an inattentive state. However, in such a situation, when the gaze cannot be detected due to a temporary event such as blinking or brightness abnormality, it is highly likely that the gaze of the occupant is not within the driving area even after the occupant finishes blinking, so it is considered appropriate to maintain the judgment that the occupant is inattentive driving. Therefore, when the cause of the inability to detect the gaze is due to a temporary event such as blinking or brightness abnormality, it is considered whether the occupant has been judged to be inattentive within the past few frames, that is, whether the occupant has been judged to be inattentive within a predetermined time from the present. Whether within the past few frames is appropriately set based on the frame rate. Since the time it takes a person to blink is about 0.10 to 0.15 seconds, it is considered whether or not the occupant has been determined to be inattentive within a number of frames equivalent to, for example, 0.12 seconds. If the occupant has been determined to be inattentive within the past few frames as a result of the consideration, the inattention state determination unit 14 determines that the occupant in the current frame is inattentive. The result of the determination is recorded in a memory (not shown).
[0027] In this way, by considering whether or not an inattention state has been determined within the past few frames, when the line of sight of a vehicle occupant in an inattentive state cannot be detected, it is possible to prevent an immediate determination that the occupant is not inattentive based on the facial direction, thereby maintaining the determination that the occupant is inattentive and improving the accuracy of cancellation when the determination that the occupant is inattentive is cancelled.
[0028] Further, the inattention state determination unit 14 determines whether or not the occupant is in an inattention state requiring an alarm based on the determination result for each frame. For example, when the determination that the occupant is in an inattention state for each frame continues for a predetermined time, for example, 2 seconds or more, the inattention state determination unit 14 presumes that the occupant is looking aside and determines that the occupant is in an inattention state requiring an alarm. When it is determined that the occupant is in an inattention state requiring an alarm, the inattention state determination unit 14 outputs information indicating that the occupant is in an inattention state requiring an alarm to a vehicle-side ECU (not shown), and a predetermined alarm is issued under the control of the vehicle-side ECU. For example, an alarm sound is generated via a speaker (not shown), and an alarm is displayed via a display or indicator (not shown).
[0029] Next, the hardware configuration of the occupant state detection device 1 will be described with reference to Fig. 2A and Fig. 2B. As an example, as shown in Fig. 2A, the occupant state detection device 1 is realized by a processing circuit 100a. The processing circuit 100a realizes the functions of the functional units included in the occupant state detection device 1. The processing circuit 100a is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these. The functions of each functional unit of the occupant state detection device 1 may be realized by separate processing circuits, or these functions may be realized collectively by one processing circuit.
[0030] As another example, as shown in Fig. 2B, the occupant state detection device 1 is realized by a processor 100b and a memory 100c. The functions of the functional units of the occupant state detection device 1 are realized by reading and executing a program stored in the memory 100c by the processor 100b. The program is realized as software, firmware, or a combination of software and firmware. Examples of the memory 100c include non-volatile or volatile semiconductor memories such as a random access memory (RAM), a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), and an electrically-EPROM (EEPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, and a DVD.
[0031] <Actions: Blink detection, dozing detection, brightness abnormality detection> Next, the operation of the occupant state detection device 1 will be described with reference to Fig. 3. In step ST101, the image acquisition unit 11 acquires from the imaging device 2 an image captured by the imaging device 2.
[0032] In step ST102, the feature information detection unit 12 detects the facial feature points, line of sight, and facial direction of the occupant from the captured image acquired by the image acquisition unit 11. The feature information detection unit 12 outputs feature information indicating the detected feature points, line of sight information indicating the detected line of sight, and facial direction information indicating the detected facial direction.
[0033] In steps ST103 to ST105, the factor estimating section 13 uses the feature information output from the feature information detecting section 12 to perform processing for estimating a factor that impedes gaze detection.
[0034] In step ST103, the factor estimation unit 13 performs the following operation on a certain frame F L , and its frame F L Frame F within the time period corresponding to the eye-closure time following L+NUsing this feature information, the degree to which the occupant's eyes are open is calculated for each frame.
[0035] In step ST104, the factor estimation unit 13 determines whether or not a blink has occurred. L Detecting eye closure for frame F L+N If an eye opening is detected for the eye, it is determined that a blink has occurred. If it is determined that a blink has occurred, the process proceeds to step ST105, and the factor estimation unit 13 sets the blink flag to ON. If it is not determined that a blink has occurred, the blink flag remains OFF as the default, and the process proceeds to step ST106.
[0036] The factor estimation unit 13 may estimate dozing instead of blinking as a factor that interferes with gaze detection. In this case, in step ST103, the factor estimation unit 13 estimates, for example, frame F L , and frame F L Frame F after the eye-closing time has elapsed L+N+1 Using the captured image of frame F, the degree of eye opening of the occupant is calculated for each frame. Then, in step ST104, the factor estimation unit 13 determines whether the occupant has fallen asleep. L Detecting eye closure for frame F L+N+1 When it is determined that the driver is dozing, the process proceeds to step ST105, where the cause estimation unit 13 sets a dozing flag to ON. When it is determined that the driver is not dozing, the dozing flag remains OFF as the default, and the process proceeds to step ST106.
[0037] In step ST103, the factor estimation unit 13 may calculate the luminance of the eye area or the like in the captured image instead of calculating the degree of eye opening. In this case, in step ST104, the factor estimation unit 13 determines whether the luminance is abnormal. If the luminance is abnormal, the process proceeds to step ST105, where the factor estimation unit 13 sets the luminance flag to ON. If the luminance is not determined to be abnormal, the luminance flag remains OFF as default, and the process proceeds to step ST106.
[0038] In step ST106, the inattention state determination unit 14 determines whether or not the line of sight detected by the feature information detection unit 12 is within the driving area. If the line of sight is not within the driving area, the process proceeds to step ST107, where the inattention state determination unit 14 sets the line of sight flag to ON. If the line of sight is within the driving area, the inattention state determination unit 14 keeps the line of sight flag at the default OFF state.
[0039] In step ST108, which is parallel to steps ST103 to ST106, the inattention state determination unit 14 determines whether or not the facial direction detected by the feature information detection unit 12 is within the driving area. If the facial direction is not within the driving area, the process proceeds to step ST109, where the inattention state determination unit 14 sets the facial direction flag to ON. If the facial direction is within the driving area, the inattention state determination unit 14 keeps the facial direction flag at the default OFF state.
[0040] The processes of steps ST110 to ST117 are all performed by the inattention state determination unit 14. In the processes of these steps, the inattention state determination unit 14 determines whether or not the occupant is in an inattention state unsuitable for driving the vehicle based on the detected line of sight and face direction, and all or part of the blink flag, drowsiness flag, and luminance flag as output cause information.
[0041] In step ST110, it is determined whether or not both the line of sight flag and the face direction flag are ON. If both are ON, the process proceeds to step ST117, and the determination result that the occupant in the captured image of the current frame is in an inattentive state is recorded in a memory (not shown). If neither are ON, the process proceeds to step ST111.
[0042] In step ST111, it is determined whether or not both the line of sight flag and the face direction flag are OFF. If both are OFF, the process proceeds to step ST116, and the determination result that the occupant in the captured image of the current frame is not in an inattentive state is recorded in a memory (not shown). If neither is OFF, the process proceeds to step ST112.
[0043] In step ST112, it is determined whether the blink flag or the luminance flag is ON. The determination may be made on the dozing flag instead of the blink flag. If the blink flag or the luminance flag is not ON, the process proceeds to step ST115. If the blink flag or the luminance flag is ON, the process proceeds to step ST113.
[0044] In step ST115, it is determined whether the line-of-sight flag is ON or not. If the line-of-sight flag is ON, the process proceeds to step ST117, and if the line-of-sight flag is OFF, the process proceeds to step ST116.
[0045] In step ST113, it is determined whether the face direction flag is ON. If the face direction flag is ON, the process proceeds to step ST117. If the face direction flag is not ON, that is, if the face direction flag is OFF, the process proceeds to step ST114.
[0046] In step ST114, it is determined whether or not the inattention state has been determined within the past few frames. For example, if the current frame is frame F L If five frames are predetermined as the number of frames, then the frame F included within the past few frames L-5 ~F L-1 It is determined whether or not the state of inattention is determined for frame F L-5 If the inattention state is determined for frame F L-5 Since frame F is within five frames, it is determined that the driver has been in an inattentive state within the past few frames. L-1 If the inattention state is determined for frame FL-1 is within five frames going back from frame F, it is determined that the state has been determined to be in an inattentive state within the past few frames. Note that the determination results for each frame are stored in a memory (not shown), and the inattentive state determination unit 14 refers to the memory (not shown) to obtain the determination results for past frames, and uses the obtained determination results in determining the current frame.
[0047] If it has been determined that the driver is in an inattentive state within the past few frames, the process proceeds to step ST117. If it has not been determined that the driver is in an inattentive state within the past few frames, the process proceeds to step ST116.
[0048] By the above processing, when cause information for the captured image of the current frame is output in step ST205 (YES in step ST112) and the face direction detected from the captured image of the current frame is facing the driving area (NO in step ST113), if the determination of whether or not the captured image of a previous frame within a predetermined number of frames from the current frame indicates an inattentive state (YES in step ST114), the inattentive state determination unit 14 determines that the occupant is in an inattentive state for the captured image of the current frame and records the determination result (step ST117).
[0049] <Glasses reflection judgment> In the above, the causes of line-of-sight detection being hindered have been described in terms of blinking, dozing, or abnormal brightness. Below, with reference to FIG. 4, the cause of line-of-sight detection being hindered will be described in terms of glasses reflection.
[0050] Figure 4 differs from Figure 3 in that it includes steps ST204 and ST205 instead of steps ST103 to ST105 in Figure 3. In accordance with this difference, Figure 4 also differs from Figure 3 in that it includes step ST212 instead of step ST112 in Figure 3. The other steps in Figure 4 are the same as those in Figure 3, so their description will be omitted.
[0051] In step ST204, the factor estimation unit 13 judges whether the gaze detected by the feature information detection unit 12 is from only one eye. If the gaze detected is from only one eye, the process proceeds to step ST205, where the factor estimation unit 13 sets the eyeglasses reflection flag to ON. If the gaze detected is not from only one eye, that is, if the gazes of both eyes are detected, the eyeglasses reflection flag remains OFF as the default, and the process proceeds to step ST106.
[0052] In step ST212, the inattention state determination unit 14 determines whether the glasses reflection flag is ON or not. If the glasses reflection flag is ON, the process proceeds to step ST113. If the glasses reflection flag is OFF, the process proceeds to step ST115.
[0053] <Additional Notes> Certain aspects of the various embodiments described above are summarized below.
[0054] (Appendix 1) The occupant state detection device (1) of Appendix 1 includes an image acquisition unit (11) configured to acquire captured images of a plurality of frames acquired by an imaging device arranged in a vehicle, a feature information detection unit (12) configured to detect the line of sight and a facial direction of an occupant of the vehicle from the captured images of the plurality of frames, a factor estimation unit (13) configured to estimate a factor that prevents detection of the line of sight of the occupant for the captured images of the plurality of frames and output information indicating the estimated factor as factor information, and an inattention state determination unit (14) configured to determine whether or not the occupant is in an inattentive state that is unsuitable for driving the vehicle, based on the detected line of sight and facial direction and all or a part of the output factor information for the captured images of each frame, and the inattention state determination unit is configured to determine whether or not the occupant is in an inattentive state for the captured image of a current frame, based on the factor information output for the captured image of the current frame, the facial direction detected from the captured image of the current frame, and a determination result of an inattentive state made for an captured image of a previous frame prior to the current frame.
[0055] (Appendix 2) The occupant status detection device of Appendix 2 is the occupant status detection device described in Appendix 1, and the determination of an inattentive state made for the captured image of the past frame is a determination of an inattentive state based on the occupant's line of sight.
[0056] (Appendix 3) The occupant status detection device of Appendix 3 is the occupant status detection device described in Appendix 1 or 2, wherein the factor estimation unit is configured to, when detecting that a blink has occurred within a predetermined eye-closure time, estimate the blinking as a factor impeding gaze detection, and output information indicating the blinking as the factor information.
[0057] (Appendix 4) The occupant status detection device of Appendix 4 is the occupant status detection device described in Appendix 1 or 2, wherein the factor estimation unit is configured, when it detects eyes closed for a period exceeding a predetermined eye closing time, to infer drowsiness as a factor impeding detection of the gaze, and to output information indicating drowsiness as the factor information.
[0058] (Appendix 5) The occupant status detection device of Appendix 5 is the occupant status detection device described in Appendix 1 or 2, wherein the factor estimation unit is configured to estimate a luminance abnormality as a factor preventing detection of the gaze when the luminance value of the captured image is equal to or lower than a predetermined first luminance value or equal to or higher than a predetermined second luminance value higher than the first luminance value, and to output information indicating the luminance abnormality as the factor information.
[0059] (Appendix 6) The occupant status detection device of Appendix 6 is the occupant status detection device described in Appendix 1 or 2, wherein the factor estimation unit is configured to estimate eyeglass reflection as a factor impeding detection of the gaze, and to output information indicating the eyeglass reflection as factor information.
[0060] (Appendix 7) The occupant state detection device of Supplementary Note 7 is the occupant state detection device according to any one of Supplementary Notes 1 to 6, wherein the past frame is within a predetermined number of frames from the current frame.
[0061] (Appendix 8) The occupant status detection device of Appendix 8 is an occupant status detection device described in any one of Appendixes 1 to 7, wherein the inattention state determination unit is configured to determine that the occupant is in an inattentive state for the captured image of the current frame when cause information is output for the captured image of the current frame and the facial direction detected from the captured image of the current frame is facing the driving area, and when the determination of whether or not the occupant is in an inattentive state made for an captured image of a past frame within a predetermined number of frames from the current frame indicates an inattentive state.
[0062] (Appendix 9) The occupant state detection method of supplementary note 9 is an occupant state detection method performed by an occupant state detection device including an image acquisition unit, a feature information detection unit, a factor estimation unit, and an inattention state determination unit, the method including the steps of: the image acquisition unit acquiring a plurality of frames of captured images acquired by an imaging device arranged in a vehicle; the feature information detection unit detecting a line of sight and a facial direction of an occupant of the vehicle from the captured images of the plurality of frames; the factor estimation unit estimating a factor that prevents detection of the line of sight of the occupant from the captured images of the plurality of frames and outputting information indicating the estimated factor as factor information; The method includes a step in which an attention state determination unit determines whether or not the occupant is in an inattentive state that is unsuitable for driving the vehicle based on all or part of the detected line of sight and facial direction and the output factor information for the captured image of each frame, and the inattentive state determination unit determines whether or not the occupant is in an inattentive state for the captured image of the current frame based on the factor information output for the captured image of the current frame, the facial direction detected from the captured image of the current frame, and the determination result of an inattentive state made for the captured image of a previous frame prior to the current frame.
[0063] It is possible to combine the embodiments, and to modify or omit each embodiment as appropriate. [Industrial Applicability]
[0064] The occupant status detection device of the present disclosure can be mounted in a vehicle and used as a device for monitoring whether or not the occupant of the vehicle is driving appropriately. [Explanation of symbols]
[0065] 1 occupant state detection device, 2 imaging device, 11 image acquisition section, 12 feature information detection section, 13 factor estimation section, 14 inattention state determination section, 100a processing circuit, 100b processor, 100c memory.
Claims
1. an image acquisition unit configured to acquire a plurality of frames of captured images acquired by an imaging device disposed in a vehicle; a feature information detection unit configured to detect a line of sight and a facial direction of an occupant of the vehicle from the captured images of the plurality of frames; a factor estimating unit configured to estimate a factor that hinders detection of the line of sight of the occupant from the captured images of the plurality of frames and output information indicating the estimated factor as factor information; an inattention state determination unit configured to determine whether or not the occupant is in an inattention state unsuitable for driving the vehicle based on the detected line of sight and face direction, and all or a part of the output cause information, for the captured image of each frame; Equipped with the inattention state determination unit is configured to, when the line of sight of the occupant cannot be detected in the captured image of the current frame, determine whether or not the occupant is in an inattention state based on the cause information outputted for the captured image of the current frame, the facial direction detected from the captured image of the current frame, and a determination result of an inattention state made for an captured image of a past frame prior to the current frame. Occupant status detection device.
2. The determination of the inattention state made for the captured image of the past frame is a determination of the inattentive state based on the line of sight of the occupant. An occupant condition detection device according to claim 1.
3. The factor estimating unit is configured to estimate the blinking as a factor that hinders gaze detection when it detects that a blink has occurred within a predetermined eye-closing time, and to output information indicating the blinking as the factor information. An occupant condition detection device according to claim 2.
4. The factor estimating unit is configured to estimate dozing as a factor that impedes gaze detection when it detects eyes closed for a period of time that exceeds a predetermined eye closing time, and to output information indicating dozing as the factor information. An occupant condition detection device according to claim 2.
5. the factor estimation unit is configured to estimate a luminance abnormality as a factor preventing detection of the line of sight when a luminance value of the captured image is equal to or lower than a predetermined first luminance value or equal to or higher than a predetermined second luminance value higher than the first luminance value, and to output information indicating the luminance abnormality as the factor information. An occupant condition detection device according to claim 2.
6. The factor estimating unit is configured to estimate eyeglass reflection as a factor that impedes gaze detection, and to output information indicating eyeglass reflection as factor information. An occupant condition detection device according to claim 2.
7. The past frame is within a predetermined number of frames from the current frame. An occupant condition detection device according to any one of claims 3 to 6.
8. The inattention state determination unit is configured to determine that the occupant is in an inattentive state with respect to the captured image of the current frame when factor information is output for the captured image of the current frame, and when a face direction detected from the captured image of the current frame is directed toward a driving area, and when a determination made for an captured image of a past frame within a predetermined number of frames from the current frame indicates an inattentive state, An occupant condition detection device according to claim 1.
9. An occupant state detection method performed by an occupant state detection device including an image acquisition unit, a feature information detection unit, a factor estimation unit, and an inattention state determination unit, The image acquisition unit acquires a plurality of frames of captured images acquired by an imaging device disposed in a vehicle; a step in which the feature information detection unit detects a line of sight and a facial direction of an occupant of the vehicle from the captured images of the plurality of frames; the factor estimating unit estimating a factor that hinders detection of the line of sight of the occupant for the captured images of the plurality of frames and outputting information indicating the estimated factor as factor information; a step of the inattention state determination unit determining whether or not the occupant is in an inattention state unsuitable for driving the vehicle based on the detected line of sight and face direction and all or a part of the output factor information for the captured image of each frame; Equipped with when the line of sight of the occupant cannot be detected in the captured image of the current frame, the inattention state determination unit determines whether or not the occupant is in an inattention state based on the cause information output for the captured image of the current frame, the facial direction detected from the captured image of the current frame, and a determination result of an inattention state made for the captured image of a previous frame prior to the current frame. Occupant status detection method.
Citation Information
Patent Citations
Projector control device
JP2011042179A
Inattentive-driving determination apparatus
JP2012014505A
Driving switching support device and driving switching support method
JP2018133032A
Visual recognition object detection device, visual recognition object detection method, and program
JP2019101520A
Monitor and monitoring method, controller and control method, and program
WO2007105792A1
