Occupant state determination device, occupant state determination system, occupant state determination method and program

Description

[Technical field]

[0001] The present disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and a program. [Background technology]

[0002] Technology has been developed that captures images of the inside of a vehicle and uses the facial direction, head position, etc. of the occupants extracted from the captured images to determine whether the occupants are losing consciousness or have other abnormalities. It also uses actions that occupants who are not actually in a bad posture may make to suppress erroneous detection of bad posture when it is not the case.

[0003] For example, the driver monitoring system of Patent Document 1 includes an abnormality diagnosis unit that determines that there is an abnormality in the driver when the absolute value of the roll angle of the driver's facial direction is maintained at or above a predetermined angle (20°, preferably 30°, more preferably 40°) for or above a predetermined time (a dozen seconds or several tens of seconds), and a change degree detection unit that detects the degree of change in the driver's face based on a facial image.Even if the abnormality diagnosis unit determines that there is an abnormality in the driver, it determines that there is no abnormality in the driver if the degree of change in the driver detected by the change degree detection unit is equal to or above a predetermined standard (for example, if the number of times the driver blinks per unit time is equal to or above a standard number (for example, once)). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2019-087018 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even though the posture is actually poor, there are cases where an occupant who is not in a bad posture may act in a manner that is possible with a non-poor posture. Examples of cases where an occupant who is not in a bad posture may act in a bad posture include cases where the occupant blinks while losing their posture, and where the occupant continues to blink after losing their posture. In such cases, the driver monitor system of Patent Document 1 may detect blinking, which is an action that a non-poor posture occupant may take, despite the posture being poor, and may determine that the occupant does not have a bad posture and that there is nothing abnormal about the occupant. Therefore, there is a problem that the posture is not detected despite the posture being poor, and the accuracy of detecting the occupant's condition is reduced.

[0006] The present disclosure has been made to solve such problems, and has an object to provide an occupant state determination device that improves the accuracy of detecting the occupant state. [Means for solving the problem]

[0007] The occupant state determination device according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image in which the face of the occupant is captured, a feature detection unit that detects at least one of the occupant's facial direction and head position, as well as facial features, using the occupant's face in the image detected by the face detection unit, a reference posture estimation unit that estimates at least one of the occupant's reference facial direction and head position as a reference posture, using at least one of the occupant's facial direction and head position detected by the feature detection unit, during a predetermined period of time, and a reference posture estimation unit that estimates at least one of the occupant's reference facial direction and head position as a reference posture, a range determination unit that performs a range determination as to whether at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the first range, the second range, or the third range set in the real space, or whether an amount of displacement of at least one of the facial direction and head position of the occupant detected by the feature detection unit from at least one of the facial direction and head position of the occupant in a reference posture falls within the first range, the second range, or the third range set in the real space; A posture deviation determination unit performs a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant corresponds to by comparing at least one of the facial direction and head position of the occupant detected by the detection unit with at least one of the facial direction and head position of the occupant in a reference posture, a face change degree detection unit detects a degree of change in the face of the occupant using the facial parts of the occupant detected by the feature detection unit, and when the posture deviation determination unit determines that the posture of the occupant corresponds to one of the plurality of posture deviation types and the face change degree detection unit detects a degree of change, the determination of the posture deviation determination unit is erroneous. and an error detection determination unit which determines that the posture of the occupant falls into one of a plurality of posture deviation types and, if the face change degree detection unit does not detect a degree of change, determines that the determination of the posture deviation determination unit is not erroneous; and if the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls into a third range, if the range determination unit determines that the amount of displacement falls into the third range, or if the error detection determination unit determines that the determination of the posture deviation determination unit is not erroneous, determines that the occupant is in an abnormal posture, andand an abnormal posture determination unit which determines that the occupant is not in an abnormal posture when the feature detection unit determines that at least one of the facial orientation and head position of the occupant detected falls within a first range, when the range determination unit determines that the displacement amount falls within the first range, when the posture deviation determination unit determines that the posture of the occupant does not fall within any of the multiple posture deviation types, or when the erroneous detection determination unit determines that the determination of the posture deviation determination unit is erroneous.

[0008] The occupant state determination system according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image in which the face of the occupant is captured, a feature detection unit that detects at least one of the occupant's facial direction and head position, as well as facial features, using the occupant's face in the image detected by the face detection unit, a reference posture estimation unit that estimates at least one of the occupant's reference facial direction and head position as a reference posture, using at least one of the occupant's facial direction and head position detected by the feature detection unit, during a predetermined period of time, and a range determination unit that performs range determination as to whether at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within a first range, a second range, or a third range set in space, or performs range determination as to whether an amount of displacement of at least one of the facial direction and head position of the occupant detected by the feature detection unit from at least one of the facial direction and head position of the occupant in a reference posture falls within a first range, a second range, or a third range set in real space; a posture deviation determination unit which performs a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant corresponds to by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in a reference posture; a face change degree detection unit which detects a degree of change in the face of the occupant using the facial parts of the occupant detected by the feature detection unit; and when the posture deviation determination unit determines that the posture of the occupant corresponds to one of the plurality of posture deviation types and the face change degree detection unit detects a degree of change, the determination of the posture deviation determination unit is erroneous. an error detection determination unit which determines that the posture of the occupant falls into one of a plurality of posture deviation types and, if the face change degree detection unit does not detect a degree of change, determines that the determination of the posture deviation determination unit is not erroneous; and if the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls into a third range, if the range determination unit determines that the amount of displacement falls into the third range, or if the error detection determination unit determines that the determination of the posture deviation determination unit is not erroneous, determines that the occupant is in an abnormal posture,an abnormal posture determination unit that determines that an occupant is not in an abnormal posture when the feature detection unit determines that at least one of the facial direction and head position of the occupant detected falls within a first range, when the range determination unit determines that the displacement falls within the first range, when the posture collapse determination unit determines that the posture of the occupant does not fall within any of a plurality of posture collapse types, or when the erroneous detection determination unit determines that the determination of the posture collapse determination unit is erroneous; an imaging device that is installed in a vehicle so as to be able to image at least the face of the occupant; and an output device that is mounted on the vehicle and outputs a warning that the posture of the occupant is abnormal, the occupant state determination device uses an image captured by the imaging device to determine whether the posture of the occupant is abnormal or not, and when it is determined that the posture of the occupant is abnormal, causes the output device to output a warning that the posture of the occupant is abnormal.

[0009] The occupant state determination method according to the present disclosure includes a step of a face detection unit detecting the face of a vehicle occupant in an image in which the face of the occupant is captured, a step of a feature detection unit detecting at least one of the occupant's facial direction and head position, and facial features, using the occupant's face in the image detected by the face detection unit, a step of a reference posture estimation unit estimating at least one of the occupant's reference facial direction and head position as a reference posture, using at least one of the occupant's facial direction and head position detected by the feature detection unit, during a predetermined period of time, and a step of a range determination unit determining a range of the occupant's facial direction and head position as a reference posture, using at least one of the occupant's facial direction and head position detected by the feature detection unit. a step of determining whether at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within a first range, a second range, or a third range set in real space, or determining whether an amount of displacement of at least one of the facial direction and head position of the occupant detected by the feature detection unit from at least one of the facial direction and head position of the occupant in a reference posture falls within a first range, a second range, or a third range set in real space; When it is determined that the displacement amount falls within the second range, the method further comprises: a step of performing a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant falls by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in a reference posture; a step of a face change degree detection unit detecting a degree of change in the face of the occupant using the facial parts of the occupant detected by the feature detection unit; and a step of an erroneous detection determination unit detecting that the posture of the occupant falls into one of the plurality of posture deviation types. a step of determining that the judgment of the posture deviation judgment unit is erroneous when the facial change degree detection unit detects a degree of change, determining that the judgment of the posture deviation judgment unit is erroneous when the facial change degree detection unit detects a degree of change, determining that the judgment of the posture deviation judgment unit is not erroneous when the facial change degree detection unit does not detect a degree of change, and determining that the judgment of the posture deviation judgment unit is not erroneous when the abnormal posture judgment unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within a third range, and determining that the amount of displacement falls within the third range when the range judgment unit determines that the amount of displacement falls within the third range.or determining that the occupant is in an abnormal posture if the erroneous detection determination unit determines that the determination by the posture deviation determination unit is not erroneous, and determining that the occupant is not in an abnormal posture if the range determination unit determines that at least one of the facial orientation and head position of the occupant detected by the feature detection unit falls within a first range, if the range determination unit determines that the amount of displacement falls within the first range, if the posture deviation determination unit determines that the posture of the occupant does not fall within any of the multiple posture deviation types, or if the erroneous detection determination unit determines that the determination by the posture deviation determination unit is erroneous.

[0010] The program according to the present disclosure includes a step of a face detection unit detecting the face of a vehicle occupant in an image in which the face of the occupant is captured, a step of a feature detection unit detecting at least one of the occupant's facial direction and head position, and facial features, using the occupant's face in the image detected by the face detection unit, a step of a reference posture estimation unit estimating at least one of the occupant's reference facial direction and head position as a reference posture, using at least one of the occupant's facial direction and head position detected by the feature detection unit, during a predetermined period of time, and a step of a range determination unit estimating at least one of the occupant's reference facial direction and head position as a reference posture, using at least one of the occupant's facial direction and head position detected by the feature detection unit. a step of performing a range determination as to whether at least one of the facial direction and head position of the occupant falls within any of a first range, a second range, and a third range set in a real space, or a range determination as to whether an amount of displacement of at least one of the facial direction and head position of the occupant detected by the feature detection unit from at least one of the facial direction and head position of the occupant in a reference posture falls within any of a first range, a second range, and a third range set in a real space; When it is determined that the displacement amount falls within the second range, the method further comprises: a step of performing a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant falls by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in a reference posture; a step of a face change degree detection unit detecting a degree of change in the face of the occupant using the facial parts of the occupant detected by the feature detection unit; and a step of an erroneous detection determination unit detecting that the posture of the occupant falls into one of the plurality of posture deviation types. a step of determining that the judgment of the posture deviation judgment unit is erroneous when the facial change degree detection unit detects a degree of change, determining that the judgment of the posture deviation judgment unit is erroneous when the facial change degree detection unit detects a degree of change, determining that the judgment of the posture deviation judgment unit is not erroneous when the facial change degree detection unit does not detect a degree of change, and determining that the judgment of the posture deviation judgment unit is not erroneous when the abnormal posture judgment unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within a third range, and determining that the amount of displacement falls within the third range when the range judgment unit determines that the amount of displacement falls within the third range.or if the erroneous detection determination unit determines that the determination by the posture error determination unit is not erroneous, determining that the occupant is in an abnormal posture, and if the range determination unit determines that at least one of the facial orientation and head position of the occupant detected by the feature detection unit falls within a first range, if the range determination unit determines that the amount of displacement falls within the first range, if the posture error determination unit determines that the posture of the occupant does not fall within any of the multiple posture error types, or if the erroneous detection determination unit determines that the determination by the posture error determination unit is erroneous, determining that the occupant is not in an abnormal posture. Effect of the Invention

[0011] According to the occupant state determination device according to the embodiment of the present disclosure, it is possible to improve the accuracy of detecting the occupant state. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration of an occupant state determination device according to a first embodiment. [Diagram 2] 4 is a diagram showing a first range, a second range, and a third range of the facial direction of an occupant in the first embodiment. FIG. [Diagram 3] 4 is a diagram showing a first range, a second range, and a third range for the head position of an occupant in the first embodiment. FIG. [Figure 4] 5 is a flowchart showing an example of the operation of the occupant state determination device according to the first embodiment. [Diagram 5] 11A and 11B are diagrams for explaining a plurality of types of poor posture. [Figure 6] 1 is a diagram illustrating an example of a hardware configuration of an occupant state determination device according to a first embodiment. [Figure 7] 1 is a diagram illustrating an example of a hardware configuration of an occupant state determination device according to a first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 The occupant state determination device according to the first embodiment determines whether or not the posture of a vehicle occupant is abnormal, based on a captured image in which at least the face of the vehicle occupant is captured. In the following first embodiment, as an example, the vehicle occupant for which the occupant state determination device determines whether or not the vehicle is in an abnormal posture is the driver of the vehicle.

[0014] FIG. 1 is a diagram showing an example of the configuration of an occupant state determination device 1 according to the first embodiment. The occupant state determination device 1 according to the first embodiment is assumed to be mounted in a vehicle. The occupant state determination device 1 is connected to an imaging device 2 and an output device 3, and the occupant state determination device 1, the imaging device 2, and the output device 3 constitute an occupant state determination system 100.

[0015] The imaging device 2 is mounted on a vehicle and installed so as to be able to capture at least an area in which the driver's face should be present. In the first embodiment, it is assumed that the imaging device 2 is installed, for example, near the center of the instrument panel in the vehicle width direction or in the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) that is installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the occupant state determination device 1. The occupant state determination device 1 determines whether or not the driver's posture is abnormal, using the captured image captured by the imaging device 2. The occupant state determination device 1 will be described in detail later. When the occupant state determination device 1 determines that the driver's posture is abnormal, it causes the output device 3 to output a warning that the driver's posture is abnormal.

[0016] The output device 3 is mounted on the vehicle and outputs a warning that the driver's posture is abnormal. The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be provided in, for example, an audio device provided in a vehicle. For example, when information for outputting a warning (hereinafter referred to as "warning information") is output from the occupant state determination device 1, the output device 3 outputs a warning sound or a voice message notifying that the driver's posture is abnormal. The output device 3 may also be, for example, a display device such as a display. For example, when warning information is output from the occupant state determination device 1, the output device 3 displays a message notifying that the driver's posture is abnormal. The output device 3 may be, for example, a turn signal, a hazard lamp, or a headlight provided on the exterior of a vehicle so as to be visible from other vehicles. For example, the occupant state determination device 1 may output warning information to the output device 3 to notify people outside the vehicle, such as occupants of other vehicles, that the driver of the vehicle is in an abnormal state. The output device 3 may be mounted on, for example, another vehicle. The occupant state determination device 1 can transmit warning information to the output device 3 mounted on the other vehicle and output a voice or display to inform the occupants of the other vehicle that the driver of the vehicle is in an abnormal state.

[0017] As shown in FIG. 1, the occupant state determination device 1 includes an image acquisition unit 11, a face detection unit 12, a feature detection unit 13, a reference posture estimation unit 14, a determination unit 15, and an output control unit 16. The feature detection unit 13 includes a face direction detection unit 131, a head position detection unit 132, a features position detection unit 133, and a detection reliability calculation unit . The determination unit 15 includes a range determination unit 151 , a posture error determination unit 152 , a face change degree detection unit 153 , an erroneous detection determination unit 154 , and an abnormal posture determination unit 155 .

[0018] The image acquisition unit 11 acquires a captured image from the imaging device 2 . The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0019] The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 . In detail, the face detection unit 12 detects the driver's facial feature points indicating the driver's facial parts by using a known image recognition technique such as edge detection for the captured image. The facial parts are the corners and corners of the eyes, the nose, the mouth, the eyebrows, the chin, etc. The driver's facial feature points are represented, for example, by coordinates on the captured image. The face detection unit 12 may also detect the driver's facial area. The driver's facial area is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's facial area is, for example, represented by the coordinates of the four corners of the smallest rectangle on the captured image. Since the installation position and the angle of view of the imaging device 2 are known in advance, even if a plurality of passengers are captured in the captured image, the face detection unit 12 can determine which area of ​​the captured image is the driver's face. For example, an area in the captured image where the driver's face may exist (hereinafter referred to as the "driver detection area") is set in advance, and the face detection unit 12 detects the driver's facial feature points and face area using a known image recognition technique in the driver detection area. The face detection unit 12 outputs information about the detected face of the driver (hereinafter referred to as "face information") to the feature detection unit 13. The face information is, for example, a captured image to which information capable of identifying the feature points and face area of ​​the driver's face is added.

[0020] The feature detection unit 13 detects features for determining the posture of an occupant, in this case, the driver. The features for determining the posture of the driver are the driver's facial direction, the driver's head position, and the driver's facial features. In the first embodiment, the driver's facial direction and head position detected by the feature detection unit 13 (hereinafter, the driver's facial direction detected by the feature detection unit 13 is referred to as the "detected facial direction", and the driver's head position detected by the feature detection unit 13 is referred to as the "detected head position") are, for example, the facial direction and head position in real space. In the first embodiment, the position of the driver's facial features detected by the feature detection unit 13 is, for example, the position of the facial features on a captured image. In the first embodiment, as an example, the driver's facial features are the driver's eyes. In the following description, it is assumed that the driver's facial features are the driver's eyes.

[0021] The face direction detection unit 131 detects the face direction of the driver in real space based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. The face direction detection unit 131 may detect the face direction of the driver using, for example, a known face direction detection technique for detecting the face direction from the captured image. The face direction of the driver is represented, for example, by at least one of a yaw angle, which is a rotation angle about an axis in the vertical direction of the vehicle, a pitch angle, which is a rotation angle about an axis in the horizontal direction of the vehicle, and a roll angle, which is a rotation angle about an axis in the longitudinal direction of the vehicle, which are predetermined references. The face direction detection unit 131 outputs information on the detected face direction (hereinafter referred to as "face direction information") to the reference attitude estimation unit 14 and the determination unit 15. The face direction information includes at least one of information on the yaw angle, the pitch angle, and the roll angle.

[0022] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. In the first embodiment, the head position of the driver in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the position of the driver's head in the captured image may be indicated, for example, by the center of the driver's face area or the center of a line connecting the driver's eye corners. In this case, the head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's face area in the captured image or the center of a line connecting the driver's eye corners as the driver's head position in real space.

[0023] Head position detection unit 132 may detect the driver's head position, for example, by using a known coordinate conversion technique that converts points on a captured image into points in real space. The driver's head position is represented, for example, by coordinates in real space. Head position detection unit 132 outputs information related to the detected driver's head position (hereinafter referred to as "head position information") to reference posture estimation unit 14 and determination unit 15. The head position information includes coordinate information of the driver's head position.

[0024] The part position detection unit 133 detects the position of the driver's eyes on the captured image based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. The face detection unit 12 detects the positions of the driver's right eye and left eye on the captured image. In the first embodiment, the position of the driver's eyes (hereinafter, the position of the driver's eyes detected by the feature detection unit 13 is referred to as the "detected eye position") is the center of a line connecting the inner corner and the outer corner of the driver's eyes on the captured image. The part position detection unit 133 may detect the position of the driver's eyes using, for example, a known image recognition technique. The position of the driver's eyes is represented, for example, by coordinates on the captured image. The part position detection unit 133 outputs information on the detected position of the driver's eyes (hereinafter, referred to as "part position information") to the determination unit 15. The part position information includes coordinate information of the position of the driver's right eye and coordinate information of the position of the driver's left eye.

[0025] The detection reliability calculation unit 134 calculates a detection reliability indicating the reliability that at least one of the detected face direction, head position, and face part is at least one of the predetermined face direction, head position, and face part. Then, the detection reliability calculation unit 134 outputs the calculated detection reliability to at least one of the reference posture estimation unit 14 and the determination unit 15. The detection reliability calculation method may be, for example, a method shown in the following document. In the following document, the detection reliability of at least one of the face direction, head position, and face part is calculated from the degree of match in matching between at least one of the face direction, head position, and face part learned in advance from a large number of face images and at least one of the face direction, head position, and face part extracted from an image of the driver. “RapidObjectDetectionusingaBoostedCascadeofSimpleFeatures”,2001,PaulViola,etal.

[0026] In addition, even if the method is not the method described in the above document, as long as it is a method that can detect a specific object corresponding to a facial part from an image and calculate the detection reliability, the detection reliability calculation unit 134 can use that method.

[0027] The reference posture estimation unit 14 estimates at least one of the driver's reference facial direction and head position as a reference posture using at least one of the detected facial direction and head position during a predetermined period (hereinafter referred to as the "reference setting period"). The reference setting period is, for example, a period from when the ignition of the vehicle is turned on until a predetermined time (for example, 10 seconds) has elapsed. Note that this is merely an example, and the reference setting period may be, for example, a predetermined time (for example, 3 seconds) when the vehicle speed is a predetermined speed (for example, 25 [km / h]) or higher and the steering angle is within a predetermined range (for example, ±20 degrees). In addition, at least one of the detected facial direction and head position is used to estimate at least one of the driver's reference facial direction and head position as a reference posture, but the reference posture may be estimated using at least one of the detected facial direction and head position and at least one of the detection reliability of the facial direction and the detection reliability of the head position. For example, the reference posture estimation unit 14 estimates the average value of at least one of the detected facial direction and the detected head position during a reference setting period as at least one of the detected facial direction and the detected head position in the driver's reference posture (hereinafter, the detected facial direction in the driver's reference posture is referred to as the "reference facial direction", and the detected head position in the driver's reference posture is referred to as the "reference head position"). The reference posture estimation unit 14 outputs information on the estimated reference posture of the driver (hereinafter referred to as "reference posture information") to the determination unit 15. The reference posture information includes at least one of a reference face direction and a reference head position. The reference face direction is represented by at least one of a yaw angle, a pitch angle, and a roll angle. The reference head position is represented, for example, by coordinates in real space. The reference posture information also includes coordinate information of the driver's head position as information on the driver's head position.

[0028] The range determination unit 151 performs a range determination as to whether at least one of the detected facial direction and the detected head position falls within a first range, a second range, or a third range set in real space, or performs a range determination as to whether the amount of displacement (hereinafter referred to as the displacement amount) by which at least one of the detected facial direction and the detected head position has been displaced from at least one of the reference facial direction and the reference head position falls within a first range, a second range, or a third range set in real space.

[0029] The first range, the second range and the third range are set in at least one of the following: the yaw angle in the direction of the driver's face, the pitch angle in the direction of the driver's face, the roll angle in the direction of the driver's face, the plane formed by the horizontal and height directions at the driver's head position, the plane formed by the horizontal and depth directions at the driver's head position, and the plane formed by the height and depth directions at the driver's head position.

[0030] Fig. 2 shows a first range, a second range, and a third range set for the yaw angle in the driver's facial direction, the pitch angle in the driver's facial direction, or the roll angle in the driver's facial direction. The first range, the second range, and the third range are set from the reference facial direction indicated by 0° in Fig. 2 in the order of increasing absolute value of the angle of the driver's facial direction, in the order of the first range, the second range, and the third range. The second range has a larger absolute value of the angle from the reference facial direction compared to the first range, and the third range has a larger absolute value of the angle from the reference facial direction compared to the second range.

[0031] FIG. 3 shows a first range, a second range, and a third range set on a plane formed by the horizontal direction and the height direction at the driver's head position, a plane formed by the horizontal direction and the depth direction at the driver's head position, or a plane formed by the height direction and the depth direction at the driver's head position. The first range, the second range, and the third range are set from the reference head position shown at the intersection of the vertical and horizontal axes in FIG. 3 in the order of increasing absolute value of the displacement amount of the driver's head position, in the order of the first range, the second range, and the third range. The second range has a larger absolute value of the displacement amount from the reference head position compared to the first range, and the third range has a larger absolute value of the displacement amount from the reference head position compared to the second range. Here, the horizontal direction, the height direction, or the depth direction at the driver's head position is assigned to the vertical axis and the horizontal axis.

[0032] The first range is set to a range in which at least one of the driver's facial direction and head position when posture collapse does not occur, which will be described later, can be located. The first range is set to be 0 to 15° from the reference facial direction or 0 to 50 mm from the reference head position, for example, in at least one of the yaw angle of the driver's facial direction, the pitch angle of the driver's facial direction, the roll angle of the driver's facial direction, the plane formed by the horizontal direction and height direction at the driver's head position, the plane formed by the horizontal direction and depth direction at the driver's head position, and the plane formed by the height direction and depth direction at the driver's head position.

[0033] The second range is set to a range in which at least one of the facial orientation and head position of both the driver who has not developed poor posture and the driver who has developed poor posture can be located. The second range is set to be 16 to 35° from the reference facial orientation or 51 to 250 mm from the reference head position in at least one of the following: the yaw angle of the driver's facial orientation, the pitch angle of the driver's facial orientation, the roll angle of the driver's facial orientation, the plane formed by the horizontal and height directions at the driver's head position, the plane formed by the horizontal and depth directions at the driver's head position, and the plane formed by the height and depth directions at the driver's head position.

[0034] The third range is set to a range in which at least one of the facial direction and head position of the driver who has lost his / her posture may be located. The third range is set to be 36 to 90° from the reference facial direction or 251 to 350 mm from the reference head position in at least one of the following: the yaw angle of the driver's facial direction, the pitch angle of the driver's facial direction, the roll angle of the driver's facial direction, the plane formed by the horizontal and height directions at the driver's head position, the plane formed by the horizontal and depth directions at the driver's head position, and the plane formed by the height and depth directions at the driver's head position.

[0035] If at least one of the detected face direction and the detected head position falls within the first range or the third range, or if the amount of displacement falls within the first range or the third range, range determination section 151 outputs the range determination result to abnormal posture determination section 155. If at least one of the detected face direction and the detected head position falls within the second range, or if the amount of displacement falls within the second range, range determination section 151 outputs the range determination result to posture error determination section 152. The range determination result includes information as to which of the first range, the second range, or the third range the detected face direction and at least one of the detected head position falls within, or information as to which of the first range, the second range, or the third range the amount of displacement falls within.

[0036] When the range determination unit 151 determines that at least one of the detected face direction and the detected head position falls within the second range, or that the displacement amount falls within the second range, the posture error determination unit 152 determines whether or not the driver's posture falls into any of a plurality of predetermined posture error types (hereinafter referred to as "posture error types") by comparing at least one of the detected face direction and the head position with at least one of the reference face direction and the reference head position. In the first embodiment, the determination by the posture error determination unit 152 of whether the driver's posture falls into any of a plurality of posture error types is also referred to as "posture error type determination". The plurality of posture deviation types include types of postures that are determined to be abnormal postures caused by the driver being in an abnormal state.

[0037] Here, FIG. 4 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining multiple types of posture imbalance. As shown in FIG. 4, there are multiple types of posture imbalance. FIG. 4 shows "prone" in which the driver falls forward and continues to have his face close to the steering wheel, "head down" in which the driver's face continues to face down, "backward lean" in which the driver's upper body is tilted backward and the face continues to face up, "backward lean" in which the driver's upper body is arched up and the face continues to face up, "head tilted to the side only" in which the driver's face continues to be tilted to the left or right, "sideways" in which the driver's upper body is tilted to the left or right and the face continues to be tilted in the same direction, and "leaning to the side" in which the driver's upper body is tilted to the left or right. In the present disclosure, the multiple pre-defined types of posture problems are "face down", "head down", "backward arch", "arched back", "head only sideways", "sideways", and "leaning" shown in Fig. 4. Information on the multiple types of posture problems is stored in a location that can be referenced by the posture problem determination unit 152.

[0038] In detail, the posture error determination unit 152 determines which of a plurality of posture error types the driver belongs to, depending on how much the detected face direction has changed compared to a reference face direction, or how much the detected head position has changed compared to a reference head position. Note that the conditions for determining which posture error type the driver belongs to when the detected face direction has changed compared to a reference face direction, or when the detected head position has changed compared to a reference head position, are set in advance and stored in the posture error determination unit 152. The posture error determination unit 152 may perform the posture error type determination using a learning device such as SVM (Support Vector Machine), for example.

[0039] When the driver's posture corresponds to any one of the posture deviation types, posture deviation determination unit 152 outputs the posture deviation type determination result to erroneous detection determination unit 154. When the driver's posture does not correspond to any one of the posture deviation types, posture deviation determination unit 152 outputs the posture deviation type determination result to abnormal posture determination unit 155. The posture deviation type determination result includes information on whether or not the driver's posture corresponds to any one of the posture deviation types.

[0040] The facial change degree detection unit 153 detects the degree of change in the driver's face using the parts of the driver's face detected by the feature detection unit 13. A change in the driver's face means that the driver moves at least a part of his / her face to change the state of the driver's face, and the degree of change in the face indicates the degree of change in the driver's face in a predetermined period. An example of a change in the driver's face is the driver blinking, and an example of a change in the driver's face is the number of times the driver blinks in a predetermined period. In the first embodiment, as an example, a change in the driver's face is the driver blinking, and the degree of change in the driver's face is the number of times the driver blinks in a predetermined period.

[0041] Specifically, the face change degree detection unit 153 detects the degree of eye opening of the driver using the position of the detected eye, and judges whether the driver has blinked. The degree of eye opening can be detected as a value obtained by, for example, detecting the position of the inner corner of the eye, the position of the outer corner of the eye, and the position of the highest point of the upper eyelid, dividing the distance between the straight line connecting the inner corner and the outer corner of the eye and the highest point of the upper eyelid by the distance between the inner corner and the outer corner of the eye to obtain the flattening of the eye, and dividing the flattening by a predetermined reference value. Note that the highest point of the upper eyelid is the point (vertex) of the upper eyelid that is farthest from the straight line connecting the outer corner and the inner corner of the eye. When the degree of eye opening falls below a predetermined threshold value TH1, it is judged that the eye is closed, when the degree of eye opening exceeds a predetermined threshold value TH2 that is greater than the predetermined threshold value TH1, it is judged that the eye is open, and when the eye opening is detected within a predetermined time after the eye closing is detected, it can be judged that "there is a blink". If an eye opening is not detected within a predetermined time after a closed eye is detected, it is determined that there is no blinking. In the above embodiment, the position of the detected eye is used to detect the degree to which the driver's eyes are open, and whether or not the driver is blinking is determined. However, the position of the detected eye and the detection reliability of the facial features may be used to detect the degree to which the driver's eyes are open, and whether or not the driver is blinking may be determined.

[0042] The facial change degree detection unit 153 counts the number of times that it determines that "blinking has occurred" (hereinafter referred to as the number of blinks) within a predetermined period, and if the number of blinks is equal to or greater than a threshold, it determines that the facial change degree has been detected, and if the number of blinks is less than the threshold, it determines that the facial change degree has not been detected.

[0043] Facial change degree detection section 153 outputs to erroneous detection determination section 154 a detection result indicating whether or not a facial change degree has been detected (hereinafter referred to as a “facial change degree detection result”).

[0044] If posture error judgment unit 152 judges that the driver's posture falls into one of a plurality of posture error types and facial change degree detection unit 153 detects the degree of change, error detection unit 154 judges that the driver's posture is not of the posture error type, i.e., the judgment of posture error judgment unit 152 is incorrect; if posture error judgment unit 152 judges that the driver's posture falls into one of a plurality of posture error types and facial change degree detection unit 153 does not detect the degree of change, error detection unit 154 judges that the driver's posture is of the posture error type, i.e., the judgment of posture error judgment unit 152 is not incorrect.

[0045] Error detection determination section 154 outputs to abnormal posture determination section 155 a determination result as to whether or not the determination by posture imbalance determination section 152 is erroneous (hereinafter referred to as an “error detection determination result”).

[0046] The abnormal posture judgment unit 155 judges whether the driver's posture is abnormal or not, using the range judgment result output by the range judgment unit 151, the posture deviation type judgment result output by the posture deviation judgment unit 152, and the erroneous detection judgment result output by the erroneous detection judgment unit 154.

[0047] The abnormal posture judgment unit 155 judges that the driver is not in an abnormal posture if the range judgment unit 151 judges that at least one of the driver's facial direction and head position detected by the feature detection unit 13 falls within a first range, if the range judgment unit 151 judges that the amount of displacement falls within the first range, if the posture error judgment unit 152 judges that the driver's posture does not fall into any of the posture error types, or if the error detection judgment unit 154 judges that the judgment of the posture error judgment unit 152 is incorrect.

[0048] When the range determination unit 151 determines that at least one of the driver's face direction and head position detected by the feature detection unit 13 falls within the third range, when the range determination unit 151 determines that the displacement amount falls within the third range, or when the erroneous detection determination unit 154 determines that the determination by the posture imbalance determination unit 152 is not erroneous, the abnormal posture determination unit 155 determines the duration of the state in which at least one of the driver's face direction and head position falls within the third range, the state in which the displacement amount falls within the third range, or the state in which the driver's posture is in a posture imbalance. For example, the abnormal posture determination unit 155 stores the acquired range determination result or posture imbalance type determination result in a storage unit (not shown) in association with information on the date and time when the range determination result or the posture imbalance type determination result was acquired. The abnormal posture determination unit 155 may determine the duration of the state in which at least one of the driver's face direction and head position falls within the third range, the state in which the displacement amount falls within the third range, or the state in which the driver's posture is in a posture imbalance from the stored range determination result or posture imbalance type determination result. The abnormal posture determination unit 155 determines that the driver is in an abnormal posture when the duration of the state in which at least one of the driver's facial direction and head position falls within a third range, the state in which the amount of displacement falls within a third range, or the driver's posture is out of alignment reaches a preset threshold value (hereinafter referred to as the "abnormal posture determination threshold value").

[0049] The abnormal posture determination unit 155 outputs to the output control unit 16 the determination result as to whether or not the driver is in an abnormal posture (hereinafter referred to as the “abnormal posture determination result”).

[0050] When the abnormal posture determination unit 155 determines that the driver is in an abnormal posture, the output control unit 16 outputs, to the output device 3, warning information for warning the driver.

[0051] A description will be given of the operation of the occupant state determination device 1 according to the embodiment 1. Fig. 5 is a flowchart for explaining the operation of the occupant state determination device 1 according to the embodiment 1. The operation of the occupant state determination device 1 is started, for example, after the ignition of the vehicle is turned on. It is preferable to perform the process of determining the occupant state every time an image is acquired from the imaging device 2, that is, for each frame.

[0052] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0053] The face detection unit 12 detects the face of the driver in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The face detection unit 12 outputs the face information to the feature detection unit 13 .

[0054] The face direction detection unit 131 detects the face direction of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The face direction detection unit 131 outputs the face direction information to the reference posture estimation unit 14 and the determination unit 15.

[0055] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and the determination unit 15.

[0056] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and the determination unit 15.

[0057] The part position detection unit 133 detects the part positions of the driver's face in real space based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The part position detection unit 133 outputs the part position information to the determination unit 15 .

[0058] The detection reliability calculation unit 134 calculates the detection reliability of at least one of the facial direction, head position, and facial features based on the face information output from the face detection unit 12 in step ST2, specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST4). The detection reliability calculation unit 134 outputs the detection reliability to at least one of the reference attitude estimation unit 14 and the determination unit 15.

[0059] If the reference posture estimation unit 14 has not completed estimating the driver's reference posture (if "NO" in step ST5), the reference posture of the driver is estimated using at least one of the facial direction and head position of the driver detected by the feature detection unit 13 during the reference setting period (as described above, hereinafter, the facial direction of the driver detected by the feature detection unit 13 is referred to as the "detected facial direction," and the head position of the driver detected by the feature detection unit 13 is referred to as the "detected head position") (step ST6).

[0060] As described above, the reference posture estimation unit 14 may estimate the reference posture using at least one of the detected face direction and head position, and at least one of the detection reliability of the face direction and the detection reliability of the head position.

[0061] The reference posture estimation unit 14 may determine whether or not estimation of the driver's reference posture has been completed by using, for example, a reference posture estimation completion flag. For example, when the reference posture estimation unit 14 estimates the driver's reference posture, it sets the reference posture estimation completion flag to "1". The initial value of the reference posture estimation completion flag is "0", and the reference posture estimation completion flag is initialized, for example, when the ignition of the occupant state determination device 1 is turned off. The reference posture estimation completion flag is stored in a location that can be referenced by the occupant state determination device 1.

[0062] If the reference posture estimation unit 14 has completed estimation of the driver's reference posture (if "YES" in step ST5), the range determination unit 151 performs a range determination as to whether at least one of the detected face direction and the detected head position falls within the first range, the second range, or the third range set in the real space, or a range determination as to whether the displacement amount falls within the first range, the second range, or the third range set in the real space (step ST7). If at least one of the detected face direction and the detected head position falls within the first range or the third range, or if the displacement amount falls within the first range or the third range, range determination unit 151 outputs the range determination result to abnormal posture determination unit 155. If at least one of the detected face direction and the detected head position falls within the second range, or if the amount of displacement falls within the second range, range determination section 151 outputs the range determination result to posture deviation determination section 152.

[0063] If the range determination unit 151 determines that at least one of the detected facial direction and the detected head position falls within the second range, or that the displacement amount falls within the second range (the case of "second range" in step ST7), the posture error determination unit 152 determines whether or not the driver's posture falls within any of a plurality of predetermined posture error types by comparing at least one of the detected facial direction and head position with at least one of the reference facial direction and reference head position (step ST8). When the driver's posture corresponds to any one of the posture error types, the posture error determination unit 152 outputs a posture error type determination result to the erroneous detection determination unit 154. When the driver's posture does not correspond to any of the posture imbalance types, the posture imbalance determination unit 152 outputs a posture imbalance type determination result to the abnormal posture determination unit 155.

[0064] If the driver's posture corresponds to any of the posture deviation types (if "YES" in step ST8), the posture deviation judgment unit 152 detects the degree of change in the driver's face using the facial parts of the driver detected by the feature detection unit 13. The face change degree detection unit 153 outputs the face change degree detection result to the erroneous detection determination unit 154 (step ST9).

[0065] Next, if the posture error judgment unit 152 judges that the driver's posture falls into one of a plurality of posture error types and the facial change degree detection unit 153 detects the degree of change, the error detection unit 154 judges that the driver's posture is not of the posture error type, and if the posture error judgment unit 152 judges that the driver's posture falls into one of a plurality of posture error types and the facial change degree detection unit 153 does not detect the degree of change, the error detection unit 154 judges that the driver's posture is of the posture error type. The erroneous detection determination section 154 outputs the erroneous detection determination result to the abnormal posture determination section 155 (step ST10).

[0066] The abnormal posture judgment unit 155 judges that the driver is not in an abnormal posture if the range judgment unit 151 judges that at least one of the driver's facial direction and head position detected by the feature detection unit 13 falls within the first range, or if the range judgment unit 151 judges that the amount of displacement falls within the first range (in the case of ``first range'' in step ST7), if the posture error judgment unit 152 judges that the driver's posture does not fall into any of the posture error types (in the case of ``NO'' in step ST8), or if the error detection judgment unit 154 judges that the driver's posture is not in a posture error, i.e., the judgment of the posture error judgment unit 152 is incorrect (in the case of ``NO'' in step ST10).

[0067] If the range determination unit 151 determines that at least one of the driver's facial direction and head position detected by the feature detection unit 13 falls within the third range, if the range determination unit 151 determines that the amount of displacement falls within the third range (in the case of ``third range'' in step ST7), or if the erroneous detection determination unit 154 determines that the driver's posture is poorly aligned, i.e., the determination of the posture deviation determination unit 152 is not erroneous (in the case of ``YES'' in step ST10), the abnormal posture determination unit 155 determines the duration of the state in which at least one of the driver's facial direction and head position falls within the third range, or the state in which the amount of displacement falls within the third range, or the state in which the driver's posture is poorly aligned (step ST11).

[0068] If the duration of at least one of the driver's facial direction and head position falls within a third range, or the amount of displacement falls within a third range, or the driver's posture is out of alignment reaches the abnormal posture determination threshold value (if "YES" in step ST11), the abnormal posture determination unit 155 determines that the driver is in an abnormal posture and causes the output device 3 to output a warning that the driver's posture is abnormal (step ST12). The abnormal posture determination unit 155 determines that the driver is not in an abnormal posture if at least one of the driver's facial direction and head position falls within the third range, if the amount of displacement falls within the third range, or if the duration of the driver's posture being out of alignment does not reach the abnormal posture determination threshold value (if "NO" in step ST11).

[0069] 6 and 7 are diagrams showing an example of a hardware configuration of the occupant state determination device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16 are realized by a processing circuit 1001. That is, the occupant state determination device 1 includes the processing circuit 1001 for performing control to determine whether the posture of the vehicle occupant is an abnormal posture based on a captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 6, or may be a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 7.

[0070] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0071] When the processing circuit is the processor 1004, the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16 are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 1005. The processor 1004 executes the functions of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16 by reading and executing the program stored in the memory 1005. That is, the occupant state determination device 1 includes a memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST12 in FIG. 6 described above. It can also be said that the programs stored in the memory 1005 cause the computer to execute the procedures or methods of the processes of the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the determination unit 15, and the output control unit 16. Here, the memory 1005 corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like.

[0072] The functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of the image acquisition unit 11 and output control unit 16 may be realized by a processing circuit 1001 as dedicated hardware, and the functions of the face detection unit 12, feature detection unit 13, reference posture estimation unit 14, and determination unit 15 may be realized by a processor 1004 reading and executing a program stored in a memory 1005.

[0073] The storage unit (not shown) is, for example, a memory 1005. The occupant state determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 2 or the output device 3.

[0074] As described above, according to the first embodiment, there are provided the face detection unit 12 that detects the face of a vehicle occupant in a captured image in which the face of the occupant is captured, the feature detection unit 13 that detects at least one of the facial direction and head position of the occupant and facial features using the face of the occupant in the captured image detected by the face detection unit 12, the reference posture estimation unit 14 that estimates at least one of the facial direction and head position of the occupant as a reference posture using at least one of the facial direction and head position of the occupant detected by the feature detection unit 13 during a predetermined period, and the reference posture estimation unit 15 that estimates at least one of the facial direction and head position of the occupant as a reference posture using at least one of the facial direction and head position of the occupant detected by the feature detection unit. a range determination unit that performs a range determination as to whether at least one of the facial orientation and head position of the occupant detected by the feature detection unit falls within a first range, a second range, or a third range set in real space, or whether an amount of displacement of at least one of the facial orientation and head position of the occupant detected by the feature detection unit from at least one of the facial orientation and head position of the occupant in a reference posture falls within a first range, a second range, or a third range set in real space; and when the range determination unit determines that at least one of the facial orientation and head position of the occupant detected by the feature detection unit falls within the second range, or the amount of displacement falls within the second range, a posture deviation determination unit that performs a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant falls by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in a reference posture; a facial change degree detection unit that detects a degree of change in the face of the occupant using the facial features of the occupant detected by the feature detection unit; and a facial change degree detection unit that performs a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant falls by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in a reference posture when the posture deviation determination unit determines that the posture of the occupant falls within the plurality of posture deviation types. an erroneous detection determination unit 154 which determines that the determination of posture collapse determination unit 152 is not erroneous when facial change degree detection unit 153 detects a degree of change and determines that at least one of the facial direction and head position of the occupant detected by feature detection unit 13 falls within a third range, when range determination unit 151 determines that the amount of displacement falls within a third range;or an abnormal posture determination unit 155 which determines that the occupant is in an abnormal posture if the erroneous detection determination unit 154 determines that the determination by the posture error determination unit 152 is not erroneous, and determines that the occupant is not in an abnormal posture if the range determination unit 151 determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit 13 falls within a first range, if the range determination unit 151 determines that the amount of displacement falls within the first range, if the posture error determination unit 152 determines that the posture of the occupant does not fall within any of the multiple posture error types, or if the erroneous detection determination unit 154 determines that the determination by the posture error determination unit 152 is erroneous.

[0075] The occupant state judgment device 1 does not use at least one of the driver's facial direction and head position detected by the feature detection unit 13 in all ranges to make a posture error type judgment in the posture error judgment unit 152 and an erroneous detection judgment in the erroneous detection judgment unit 154, or use the displacement amount to make a posture error type judgment in the posture error judgment unit 152 and an erroneous detection judgment in the erroneous detection judgment unit 154, but rather makes a range judgment in the range judgment unit 151 as to whether at least one of the driver's facial direction and head position detected by the feature detection unit 13 falls into the first range, the second range, or the third range, or makes a range judgment as to whether the displacement amount falls into the first range, the second range, or the third range, and changes the destination to which the range judgment result is output depending on the range judgment result.

[0076] In particular, when the range determination unit 151 determines that at least one of the driver's facial direction and head position detected by the feature detection unit 13 falls within the second range, or when it determines that the displacement amount falls within the second range, the range determination unit 151 uses at least one of the driver's facial direction and head position detected by the feature detection unit 13 to perform a posture correction type determination in the posture correction determination unit 152 and a misdetection determination in the misdetection determination unit 154, or uses the displacement amount to perform a posture correction type determination in the posture correction determination unit 152 and a misdetection determination in the misdetection determination unit 154.

[0077] On the other hand, when the range determination unit 151 determines that at least one of the driver's face direction and head position detected by the feature detection unit 13 falls within the third range, or when the range determination unit 151 determines that the displacement amount falls within the third range, it outputs the result to the abnormal posture determination unit 155, rather than performing posture deviation type determination in the posture deviation determination unit 152 and erroneous detection determination in the erroneous detection determination unit 154. When the duration of the state in which at least one of the driver's face direction and head position falls within the third range, or the state in which the displacement amount falls within the third range, reaches the abnormal posture determination threshold, the abnormal posture determination unit 155 determines that the driver is in an abnormal posture.

[0078] Therefore, the occupant state determination device 1 suppresses erroneous detection of posture imbalance in which the driver's posture is determined to be abnormal even though it is not abnormal in the second range, which is a range in which at least one of the face direction and head position of both the driver who has not experienced posture imbalance and the driver who has experienced posture imbalance can be located. Furthermore, even if there is a change in the driver's face in the third range, which is a range in which at least one of the face direction and head position of the driver who has experienced posture imbalance can be located, that is, even if the driver's posture is determined to be abnormal even if the driver blinks or continues to blink after the posture imbalance. Therefore, it is possible to suppress non-detection of posture imbalance in which the driver's posture is determined to be abnormal even though it is abnormal, and to improve the detection accuracy of the occupant state.

[0079] In the above-described first embodiment, the occupant state determination device 1 assumes that once the reference posture estimation unit 14 estimates the reference posture of the driver, the reference posture is not reviewed until, for example, the ignition of the vehicle is turned off. However, this is merely an example, and for example, the reference posture estimation unit 14 may constantly monitor the reference posture, and if the reference posture has changed within a range in which it can be said that the driver's posture has not collapsed, the reference posture may be updated.

[0080] In the above-described first embodiment, the reference posture estimation unit 14 estimates the reference posture of the driver, but this is merely an example. For example, an administrator or the like may set a reference posture in advance using the face direction, head position, and positions of facial parts such as eyes of a typical driver, and store the reference posture in a location that the determination unit 15 can refer to. In this case, the occupant state determination device 1 may be configured not to include the reference posture estimation unit 14. However, the occupant state determination device 1 can set a reference posture more suited to an individual if the reference posture estimation unit 14 estimates the reference posture, compared to a configuration not including the reference posture estimation unit 14. As a result, the occupant state determination device 1 can more accurately determine whether or not the driver is in an abnormal posture.

[0081] In addition, in the above embodiment 1, range determination unit 151 determines whether at least one of the detected face direction and the detected head position falls within the first range, the second range, or the third range set in real space, or whether the displacement amount falls within the first range, the second range, or the third range set in real space, but range determination unit 151 may determine the range by machine learning.

[0082] In the above embodiment 1, the vehicle occupant for which the occupant state determination device 1 determines whether or not the vehicle is in an abnormal posture is the driver of the vehicle, but this is merely an example. The occupant state determination device 1 can determine whether or not the vehicle occupant is in an abnormal posture of any vehicle occupant other than the driver.

[0083] In addition, in the above embodiment 1, the occupant state judgment device 1 is an in-vehicle device mounted in a vehicle, and the image acquisition unit 11, the face detection unit 12, the feature detection unit 13, the reference posture estimation unit 14, the judgment unit 15, and the output control unit 16 are provided in the in-vehicle device. Without being limited to this, some of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, judgment unit 15, and output control unit 16 may be mounted on the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, so that a system is formed by the in-vehicle device and the server. Moreover, the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, determination unit 15, and output control unit 16 may all be provided in the server. [Explanation of symbols]

[0084] 1 Occupant state determination device, 2 Imaging device, 3 Output device, 5 Steering mechanism, 6 Brake / drive mechanism, 11 Image acquisition unit, 12 Face detection unit, 13 Feature detection unit, 131 Face direction detection unit, 132 Head position detection unit, 133 Part position detection unit, 134 Detection reliability calculation unit, 14 Reference posture estimation unit, 15 Determination unit, 151 Range determination unit, 152 Posture collapse determination unit, 153 Face change degree detection unit, 154 False detection determination unit, 155 Abnormal posture determination unit, 16 Output control unit, 100 Occupant state determination system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory

Claims

1. a face detection unit that detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit that detects at least one of a facial direction and a head position of the occupant and facial features by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit that estimates at least one of a face direction and a head position of the occupant as a reference posture by using at least one of a face direction and a head position of the occupant detected by the feature detection unit during a predetermined period; a range determination unit that performs range determination as to whether at least one of a facial direction and a head position of the occupant detected by the feature detection unit falls within a first range, a second range, or a third range set in a real space, or performs range determination as to whether an amount of displacement of at least one of a facial direction and a head position of the occupant detected by the feature detection unit from at least one of a facial direction and a head position of the occupant in the reference posture falls within the first range, the second range, or the third range set in the real space; a posture imbalance determination unit which performs a posture imbalance type determination as to which of a plurality of posture imbalance types the posture of the occupant falls into by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in the reference posture when the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the second range or when the range determination unit determines that the displacement amount falls within the second range; a face change degree detection unit that detects a change degree of the face of the occupant using the facial features of the occupant detected by the feature detection unit; an erroneous detection determination unit which, when the posture deviation determination unit determines that the posture of the occupant falls under any of the plurality of posture deviation types and the facial change degree detection unit detects the degree of change, determines that the determination of the posture deviation determination unit is erroneous, and, when the posture deviation determination unit determines that the posture of the occupant falls under any of the plurality of posture deviation types and the facial change degree detection unit does not detect the degree of change, determines that the determination of the posture deviation determination unit is not erroneous; an abnormal posture determination unit which determines that the occupant is in an abnormal posture when the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the third range, when the range determination unit determines that the amount of displacement falls within the third range, or when the erroneous detection determination unit determines that the determination of the posture collapse determination unit is not erroneous, and which determines that the occupant is not in the abnormal posture when the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the first range, when the range determination unit determines that the amount of displacement falls within the first range, when the posture collapse determination unit determines that the posture of the occupant does not fall within any of the multiple posture collapse types, or when the erroneous detection determination unit determines that the determination of the posture collapse determination unit is erroneous; An occupant state determination device comprising:

2. the facial orientation of the occupant is represented by at least one of a yaw angle, which is a rotation angle about an axis in a vertical direction of the vehicle, a pitch angle, which is a rotation angle about an axis in a horizontal direction of the vehicle, and a roll angle, which is a rotation angle about an axis in a front-rear direction of the vehicle, which are predetermined references; The head position of the occupant is obtained by converting a point arbitrarily set on the face of the occupant in the captured image detected by the face detection unit into a point in the real space. The occupant condition determination device according to claim 1 .

3. the first range, the second range, and the third range are set in at least one of the yaw angle in the direction of the face of the occupant, the pitch angle in the direction of the face of the occupant, the roll angle in the direction of the face of the occupant, a plane formed by a horizontal direction and a height direction at the head position of the occupant, a plane formed by a horizontal direction and a depth direction at the head position of the occupant, and a plane formed by a height direction and a depth direction at the head position of the occupant, the first range, the second range, and the third range are each centered on at least one of a facial direction and a head position of the occupant in the reference posture, and are set in the order of the first range, the second range, and the third range from at least one of a facial direction and a head position of the occupant in the reference posture. The occupant condition determining device according to claim 2 .

4. the second range has a larger absolute value of at least one of an angle and a displacement amount from at least one of a facial direction and a head position of the occupant in the reference posture, compared to the first range; The occupant state determination device according to claim 3 , wherein the third range has a larger absolute value of at least one of an angle and a displacement amount from at least one of a facial direction and a head position of the occupant in the reference posture than the second range.

5. the degree of change in the occupant's face is the number of times the occupant blinks in a predetermined period of time; the facial change degree detection unit counts the number of blinks of the occupant during a predetermined period, and when the number of blinks is equal to or greater than a threshold, the facial change degree detection unit determines that the facial change degree has been detected, and when the number of blinks is less than the threshold, the facial change degree detection unit determines that the facial change degree has not been detected. The occupant condition determination device according to claim 1 .

6. The range determination unit determines, by machine learning, whether at least one of the facial orientation and the head position of the occupant detected by the feature detection unit falls within the first range, the second range, or the third range, or whether the displacement amount falls within the first range, the second range, or the third range. The occupant condition determination device according to claim 1 .

7. an output control unit that outputs warning information for warning the occupant when the abnormal posture determination unit determines that the occupant is in the abnormal posture; The occupant condition determination device according to claim 1 , further comprising:

8. 8. An abnormal posture determination system comprising: the occupant state determination device according to claim 1; an imaging device installed in the vehicle so as to be able to image at least a face of the occupant; and an output device mounted in the vehicle and configured to output a warning that the posture of the occupant is the abnormal posture, The occupant state determination device determines whether or not the posture of the occupant is the abnormal posture by using the captured image captured by the imaging device, and when it is determined that the posture of the occupant is the abnormal posture, causes the output device to output a warning that the posture of the occupant is the abnormal posture. Occupant status determination system.

9. A face detection unit detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit detecting at least one of a facial direction and a head position of the occupant, and facial features, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit estimating, during a predetermined period, at least one of a face direction and a head position of the occupant as a reference posture, using at least one of a face direction and a head position of the occupant detected by the feature detection unit; a range determination unit performing a range determination as to whether at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within a first range, a second range, or a third range set in a real space, or a range determination as to whether an amount of displacement of at least one of the facial direction and head position of the occupant detected by the feature detection unit from at least one of the facial direction and head position of the occupant in the reference posture falls within the first range, the second range, or the third range set in the real space; a posture imbalance determination unit, when the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the second range or when the range determination unit determines that the displacement amount falls within the second range, performs a posture imbalance type determination as to which of a plurality of posture imbalance types the posture of the occupant falls by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in the reference posture; a facial change degree detection unit detecting a degree of change in the face of the occupant using the facial features of the occupant detected by the feature detection unit; a step of an erroneous detection determination unit determining that the determination of the posture deviation determination unit is erroneous when the posture deviation determination unit determines that the posture of the occupant falls under any of the plurality of posture deviation types and the facial change degree detection unit detects the degree of change, and determining that the determination of the posture deviation determination unit is not erroneous when the posture deviation determination unit determines that the posture of the occupant falls under any of the plurality of posture deviation types and the facial change degree detection unit does not detect the degree of change; an abnormal posture determination unit determining that the occupant is in an abnormal posture if the range determination unit determines that at least one of the facial orientation and head position of the occupant detected by the feature detection unit falls within the third range, if the range determination unit determines that the amount of displacement falls within the third range, or if the erroneous detection determination unit determines that the determination of the posture collapse determination unit is not erroneous, and determining that the occupant is not in the abnormal posture if the range determination unit determines that at least one of the facial orientation and head position of the occupant detected by the feature detection unit falls within the first range, if the range determination unit determines that the amount of displacement falls within the first range, if the posture collapse determination unit determines that the posture of the occupant does not fall within any of the multiple posture collapse types, or if the erroneous detection determination unit determines that the determination of the posture collapse determination unit is erroneous; An occupant state determination method comprising:

10. A face detection unit detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit detecting at least one of a facial direction and a head position of the occupant, and facial features, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit estimating, during a predetermined period, at least one of a face direction and a head position of the occupant as a reference posture, using at least one of a face direction and a head position of the occupant detected by the feature detection unit; a range determination unit performing a range determination as to whether at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within a first range, a second range, or a third range set in a real space, or a range determination as to whether an amount of displacement of at least one of the facial direction and head position of the occupant detected by the feature detection unit from at least one of the facial direction and head position of the occupant in the reference posture falls within the first range, the second range, or the third range set in the real space; a posture imbalance determination unit, when the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the second range or when the range determination unit determines that the displacement amount falls within the second range, performs a posture imbalance type determination as to which of a plurality of posture imbalance types the posture of the occupant falls by comparing at least one of the facial direction and head position of the occupant detected by the feature detection unit with at least one of the facial direction and head position of the occupant in the reference posture; a facial change degree detection unit detecting a degree of change in the face of the occupant using the facial features of the occupant detected by the feature detection unit; a step of an erroneous detection determination unit determining that the determination of the posture deviation determination unit is erroneous when the posture deviation determination unit determines that the posture of the occupant falls under any of the plurality of posture deviation types and the facial change degree detection unit detects the degree of change, and determining that the determination of the posture deviation determination unit is not erroneous when the posture deviation determination unit determines that the posture of the occupant falls under any of the plurality of posture deviation types and the facial change degree detection unit does not detect the degree of change; an abnormal posture determination unit determining that the occupant is in an abnormal posture if the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the third range, if the range determination unit determines that the amount of displacement falls within the third range, or if the erroneous detection determination unit determines that the determination of the posture collapse determination unit is not erroneous; and determining that the occupant is not in the abnormal posture if the range determination unit determines that at least one of the facial direction and head position of the occupant detected by the feature detection unit falls within the first range, if the range determination unit determines that the amount of displacement falls within the first range, if the posture collapse determination unit determines that the posture of the occupant does not fall into any of the multiple posture collapse types, or if the erroneous detection determination unit determines that the determination of the posture collapse determination unit is erroneous.

Images