Passenger state determination device, passenger state determination system, passenger state determination method and program

Abstract

A passenger state determination device (1) according to the present disclosure is equipped with: a face detection unit (12); a feature detection unit (13) for detecting a detected passenger posture; a reference posture estimation unit (14) which classifies the detected passenger posture by using a first threshold, and estimates a mode value as the reference posture, which is the detected passenger posture which is most frequently detected by the feature detection unit from among the postures which are obtained during a pre-set interval and which comprise a detected passenger posture which can be viewed as the reference posture of the passenger and a detected passenger posture which can be viewed as a change in the posture of the passenger from the reference posture; a reference posture update determination unit (17) for determining whether or not to update the reference posture by comparing the difference between a first reference posture and a second reference posture with a second threshold; a posture collapse determination unit (18) for determining the type of posture collapse; and an abnormal posture determination unit (19) for determining whether or not the posture of the passenger is abnormal.

Description

Occupant state determination device, occupant state determination system, occupant state determination method and program

[0001] The present disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and a program.

[0002] A technology has been developed that takes an image of the inside of a vehicle, and uses the facial direction, head position, etc. of the occupant extracted from the captured image to determine whether the occupant has lost consciousness or other abnormalities, and updates the reference posture that serves as the basis for the occupant's facial direction, head position, etc. if the occupant changes their posture intentionally or due to habit (hereinafter referred to as occupant posture change) due to long periods of driving or changes in the driving environment (changes in weather or time of day).

[0003] For example, the information processing device disclosed in Patent Document 1 calculates the deviation between the driver's facial direction and a reference facial direction. If the deviation is within a first range but outside a second range, the device associates information indicating the deviation with detection information about the facial direction of the image and stores the information in a facial direction memory unit. If images showing deviations from the reference in a certain direction (approximately the same direction) are detected at a predetermined frequency (or ratio), the device performs processing to change the reference facial direction so that the state of the facial direction deviated in the certain direction is more likely to be determined as the driver's face facing forward. When the occupant's posture changes, in other words, when images showing deviations from the reference in a certain direction (approximately the same direction) are detected at a predetermined frequency (or ratio), the device updates the reference facial direction of the occupant. Here, the first range is set to various ranges depending on the purpose of monitoring. The first range is, for example, a range determined by an inattentiveness judgment angle, a concentration / fatigue level judgment angle, etc. The second range is an angle range narrower than the first range and is determined by the direction in which the face is considered to be facing forward.

[0004] JP 2019-088522 A

[0005] In reality, even if the occupant's posture is in the reference posture and the occupant does not change posture from the reference posture, due to vehicle vibrations, etc., at least one of the occupant's facial direction and head position may move back and forth between at least one of the angle and position when the occupant is in the reference posture and at least one of the angle and position when the occupant changes posture (hereinafter referred to as Situation A).

[0006] Therefore, because the occupant's posture is in the reference posture, detection frequency A may be higher than detection frequency B. Also, during a predetermined period, both the detection frequency (hereinafter referred to as detection frequency A) of at least one of the occupant's facial orientation (hereinafter referred to as facial orientation A) and head position, which can be considered to be in the reference posture, and the detection frequency (hereinafter referred to as detection frequency B) of at least one of the occupant's facial orientation (hereinafter referred to as facial orientation B), which can be considered to have changed from the reference posture, may be higher than a predetermined frequency. In other words, even though detection frequency A is higher than detection frequency B, detection frequency B may be higher than a predetermined frequency.

[0007] Here, in the case of situation A, in Patent Document 1, both the detection frequency of the face orientation of the image corresponding to a deviation within the second range corresponding to face orientation A (hereinafter referred to as detection frequency A') and the detection frequency of the face orientation of the image corresponding to a deviation outside the second range corresponding to face orientation B (hereinafter referred to as detection frequency B') may be greater than or equal to a predetermined frequency.

[0008] However, in Patent Document 1, first, it is determined whether the deviation satisfies the condition of being within a first range and outside a second range, and when the facial orientation of an image associated with the deviation that satisfies the condition is detected at a predetermined frequency, the standard for the facial orientation of the occupant is updated. Therefore, only the detection frequency of the facial orientation of an image corresponding to the deviation that satisfies the condition of being within the first range and outside the second range is used to determine whether the standard for the facial orientation needs to be changed.

[0009] Therefore, in Patent Document 1, in the case of situation A, even though detection frequency A' is higher than detection frequency B', if detection frequency B' is equal to or higher than a predetermined frequency, detection frequency A' is excluded from the target and only detection frequency B' becomes the target, so even though the actual face direction standard is the face direction having detection frequency A', processing is performed to change the face direction standard to the face direction having detection frequency B'.

[0010] In other words, when attempting to update the reference posture using Patent Document 1, even if the occupant is in the reference posture, it may be determined that the occupant has changed their posture from the reference posture, and the occupant's reference posture may be updated. This makes it difficult to accurately calculate the amount of change from the reference posture used to determine whether or not the occupant is in a state of posture collapse. As a result, even if a state of posture collapse has occurred, the amount of change cannot be accurately calculated, resulting in an undetected state of posture collapse being determined not to be a state of posture collapse, which reduces the accuracy of detecting the occupant's state.

[0011] The present disclosure has been made to solve such problems, and aims to provide an occupant state determination device that improves the accuracy of detecting the occupant state.

[0012] The occupant state determination device according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image in which the face of the occupant is captured, and a feature detection unit that uses the occupant's face in the image detected by the face detection unit to detect a detected occupant posture, which is at least one of the occupant's facial orientation and head position, and when the detected occupant posture detected by the feature detection unit is less than a first threshold, the detected occupant posture can be considered to be a reference posture, which is at least one of the occupant's facial orientation and head position serving as a reference for the occupant, and when the detected occupant posture is equal to or greater than the first threshold, the detected occupant posture can be considered to be a detected occupant posture in which the occupant has changed posture from the reference posture, and the detected occupant posture is determined based on a most frequent detected occupant posture detected by the feature detection unit among the detected occupant postures in which the occupant can be considered to be the reference posture and the detected occupant postures in which the occupant has changed posture from the reference posture, which are acquired over a predetermined period. a reference posture estimation unit that estimates a first reference posture as a posture; a reference posture update determination unit that updates the first reference posture to the second reference posture if a difference between a first reference posture, which is a reference posture estimated using the most frequent value in the captured image, and a second reference posture, which is a reference posture estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and does not update the first reference posture if the difference is less than the second threshold; a posture collapse determination unit that determines which of a plurality of posture collapse types the occupant's posture falls into by comparing the detected occupant posture with the first reference posture; and an abnormal posture determination unit that determines that the occupant has an abnormal posture if the posture collapse determination unit determines that the occupant's posture falls into any of the plurality of posture collapse types, and determines that the occupant does not have an abnormal posture if the posture collapse determination unit determines that the occupant's posture does not fall into any of the plurality of posture collapse types.

[0013] The occupant state determination system according to the present disclosure includes a face detection unit that detects the face of a vehicle occupant in an image captured of the face of the occupant; a feature detection unit that uses the occupant's face in the image detected by the face detection unit to detect a detected occupant posture, which is at least one of the occupant's facial orientation and head position; and a feature detection unit that, when the detected occupant posture detected by the feature detection unit is less than a first threshold, determines that the occupant posture can be considered to be a reference posture, which is at least one of the occupant's facial orientation and head position, which are reference postures for the occupant; and, when the detected occupant posture is equal to or greater than the first threshold, determines that the occupant has changed from the reference posture. A reference posture estimation unit estimates, as a detected occupant posture that can be considered to have changed, a most frequent value that is a detected occupant posture that is most frequently detected by a feature detection unit among detected occupant postures that can be considered to be a reference posture of the occupant acquired during a predetermined period and detected occupant postures that can be considered to have changed from the reference posture, and a difference between a first reference posture that is a reference posture estimated using the most frequent value in a captured image and a second reference posture that is a reference posture estimated using the most frequent value in a captured image different from the captured image. a reference attitude update determination unit that updates the first reference attitude to the second reference attitude when the difference is equal to or greater than a second threshold, and does not update the first reference attitude when the difference is less than the second threshold; a posture collapse determination unit that performs posture collapse type determination as to which of a plurality of posture collapse types the posture of the occupant corresponds by comparing the detected occupant attitude with the first reference attitude; and a posture collapse determination unit that determines that the posture of the occupant corresponds to which of a plurality of posture collapse types the posture of the occupant corresponds to when the posture collapse determination unit determines that the posture of the occupant corresponds to which of the plurality of posture collapse types the posture of the occupant corresponds to when the posture collapse determination unit determines that the posture of the occupant corresponds to an occupant state determination device including an abnormal posture determination unit that determines that the occupant is not in an abnormal posture if the occupant is in an abnormal posture; an imaging device installed in the vehicle so as to be able to capture an image of at least the face of the occupant; and an output device mounted in the vehicle that outputs a warning that the occupant's posture is abnormal, wherein the occupant state determination device determines whether the occupant's posture is abnormal using the image captured by the imaging device, and if it determines that the occupant's posture is abnormal, causes the output device to output a warning that the occupant's posture is abnormal.

[0014] The occupant state determination method according to the present disclosure includes a step in which a face detection unit detects the face of a vehicle occupant in an image in which the face of the occupant is captured; a step in which a feature detection unit detects a detected occupant posture, which is at least one of a facial orientation and a head position of the occupant, using the occupant's face in the image detected by the face detection unit; and a reference posture estimation unit, when the detected occupant posture detected by the feature detection unit is less than a first threshold, determines the detected occupant posture as a position in which the occupant can be considered to be a reference posture, which is at least one of a facial orientation and a head position serving as a reference for the occupant, when the detected occupant posture is equal to or greater than the first threshold, determines the detected occupant posture as a position in which the occupant can be considered to have changed from the reference posture, and determines a most frequent detected occupant posture that is detected by the feature detection unit most frequently among the detected occupant postures in which the occupant can be considered to be the reference posture and the detected occupant postures in which the occupant can be considered to have changed from the reference posture, which are acquired over a predetermined period. as a reference posture; a reference posture update determination unit updating the first reference posture to the second reference posture if a difference between a first reference posture, which is a reference posture estimated using the most frequent value in the captured image, and a second reference posture, which is a reference posture estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and not updating the first reference posture if the difference is less than the second threshold; a posture collapse determination unit comparing the detected occupant posture with the first reference posture to determine which of a plurality of posture collapse types the occupant posture falls into; and a step in which the abnormal posture determination unit determines that the occupant posture is abnormal if the posture collapse determination unit determines that the occupant posture falls into any of the plurality of posture collapse types, and determines that the occupant posture is not abnormal if the posture collapse determination unit determines that the occupant posture does not fall into any of the plurality of posture collapse types.

[0015] The program according to the present disclosure includes a step in which a face detection unit detects the face of a vehicle occupant in an image in which the face of the occupant is captured; a step in which a feature detection unit detects a detected occupant posture, which is at least one of the occupant's facial orientation and head position, using the occupant's face in the image detected by the face detection unit; and a reference posture estimation unit, when the detected occupant posture detected by the feature detection unit is less than a first threshold, determines the detected occupant posture as a position in which the occupant can be considered to be a reference posture, which is at least one of the occupant's facial orientation and head position serving as a reference for the occupant, when the detected occupant posture is equal to or greater than the first threshold, determines the detected occupant posture as a position in which the occupant can be considered to have changed from the reference posture, and determines a mode, which is the detected occupant posture detected most frequently by the feature detection unit, as the reference posture among the detected occupant postures that can be considered to be the reference posture and the detected occupant postures that can be considered to have changed from the reference posture, which are acquired over a predetermined period. a reference posture update determination unit updating the first reference posture to the second reference posture if a difference between a first reference posture, which is a reference posture estimated using the most frequent value in the captured image, and a second reference posture, which is a reference posture estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and not updating the first reference posture if the difference is less than the second threshold; a posture collapse determination unit comparing the detected occupant posture with the first reference posture to determine which of a plurality of posture collapse types the occupant posture falls into; and a step in which the abnormal posture determination unit determines that the occupant posture is abnormal if the posture collapse determination unit determines that the occupant posture falls into any of the plurality of posture collapse types, and determines that the occupant posture is not abnormal if the posture collapse determination unit determines that the occupant posture does not fall into any of the plurality of posture collapse types.

[0016] According to the occupant state determination device according to the embodiment of the present disclosure, it is possible to improve the accuracy of detecting the occupant state.

[0017] FIG. 1 is a diagram showing an example of the configuration of an occupant state determination device according to embodiment 1. FIG. 2 is a diagram showing the relationship between a detected driver posture according to embodiment 1 and the detection frequency of each value of the detected driver posture in a predetermined period. FIG. 3 is a diagram showing the relationship between a detected driver posture according to embodiment 1 and the detection frequency of each value of the detected driver posture in a predetermined period. FIG. 4 is a diagram showing the relationship between a detected driver posture according to embodiment 1 and the detection frequency of each value of the detected driver posture in a predetermined period. FIG. 5 is a diagram for explaining a plurality of posture imbalance types. FIG. 6 is a flowchart showing an example of the operation of an occupant state determination device according to embodiment 1. FIG. 7 is a diagram showing an example of the hardware configuration of an occupant state determination device according to embodiment 1.

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1. An occupant state determination device according to embodiment 1 determines whether the posture of a vehicle occupant is abnormal based on an image in which at least the face of the vehicle occupant is captured. In the following embodiment 1, as an example, the vehicle occupant whose posture is to be determined by the occupant state determination device as being abnormal is the vehicle driver.

[0019] FIG. 1 is a diagram showing an example of the configuration of an occupant state determination device 1 according to embodiment 1. The occupant state determination device 1 according to embodiment 1 is assumed to be mounted on a vehicle. The occupant state determination device 1 is connected to an imaging device 2 and an output device 3, and the occupant state determination device 1, the imaging device 2, and the output device 3 together constitute an occupant state determination system 100. When the driver intentionally or habitually changes their posture due to long-term driving or changes in the driving environment (changes in weather or time of day), the occupant state determination device 1 updates the reference posture, which is the driver's reference facial orientation and head position.

[0020] The imaging device 2 is mounted on a vehicle and is installed so as to be able to capture an image of at least the area where the driver's face should be present. In the first embodiment, the imaging device 2 is assumed to be installed, for example, near the center of the instrument panel in the vehicle width direction or on the center console. For example, the imaging device 2 may be shared with a so-called DMS (Driver Monitoring System) installed for the purpose of monitoring the interior of the vehicle. The imaging device 2 is a visible light camera or an infrared camera. The imaging device 2 outputs the captured image to the occupant state determination device 1. The occupant state determination device 1 determines whether the driver's posture is abnormal using the captured image captured by the imaging device 2. Details of the occupant state determination device 1 will be described later. If the occupant state determination device 1 determines that the driver's posture is abnormal, the occupant state determination device 1 causes the output device 3 to output a warning indicating that the driver's posture is abnormal.

[0021] The output device 3 is mounted on the vehicle and outputs a warning that the driver's posture is abnormal. The output device 3 is, for example, an audio output device such as a speaker. The output device 3 may be provided, for example, in an audio device provided in the vehicle. For example, when information for outputting a warning (hereinafter referred to as "warning information") is output from the occupant state determination device 1, the output device 3 outputs a warning sound or a voice message notifying that the driver's posture is abnormal. The output device 3 may also be, for example, a display device such as a display. For example, when the warning information is output from the occupant state determination device 1, the output device 3 displays a message notifying that the driver's posture is abnormal. The output device 3 may also be, for example, a turn signal, hazard lamp, or headlight provided on the exterior of the vehicle so as to be visible from other vehicles. For example, the occupant state determination device 1 may output warning information to the output device 3 to notify a person outside the vehicle, such as an occupant of another vehicle, that the driver of the vehicle is in an abnormal state. The output device 3 may be mounted on, for example, another vehicle. The occupant state determination device 1 can transmit warning information to the output device 3 mounted on the other vehicle and output a voice or a display to notify the occupants of the other vehicle that the driver of the vehicle is in an abnormal state.

[0022] 1 , the occupant state determination device 1 includes an image acquisition unit 11, a face detection unit 12, a feature detection unit 13, a reference posture estimation unit 14, a reference posture storage unit 15, a reference posture candidate storage unit 16, a reference posture update determination unit 17, a posture collapse determination unit 18, an abnormal posture determination unit 19, and an output control unit 20. The feature detection unit 13 includes a face direction detection unit 131, a head position detection unit 132, and a detection reliability calculation unit 133.

[0023] The image acquisition unit 11 acquires a captured image from the imaging device 2. The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0024] The face detection unit 12 detects the driver's face in the captured image acquired by the image acquisition unit 11. Specifically, the face detection unit 12 detects the driver's facial feature points, which indicate the driver's facial features, using a known image recognition technique, such as edge detection, on the captured image. The facial features include the corners and corners of the eyes, the nose, the mouth, the eyebrows, and the chin. The driver's facial feature points are represented, for example, by coordinates on the captured image. The face detection unit 12 may also detect the driver's facial region. The driver's facial region is, for example, the smallest rectangle that encloses the outline of the driver's face. The driver's facial region is represented, for example, by the coordinates of the four corners of the smallest rectangle on the captured image. Because the installation position and angle of view of the imaging device 2 are known in advance, even if multiple occupants are captured in the captured image, the face detection unit 12 can determine which area of ​​the captured image is the driver's face. For example, an area in the captured image where the driver's face may be present (hereinafter referred to as the "driver detection area") is set in advance, and the face detection unit 12 detects the driver's facial feature points and facial area in the driver detection area using a known image recognition technique. The face detection unit 12 outputs information about the detected driver's face (hereinafter referred to as "face information") to the feature detection unit 13. The face information is, for example, a captured image to which information capable of identifying the driver's facial feature points and facial area is added.

[0025] The feature detection unit 13 detects features for determining the posture of an occupant, in this case, the driver. The feature for determining the posture of the driver is at least one of the driver's facial orientation and the driver's head position (hereinafter referred to as the detected driver posture).

[0026] The face direction detection unit 131 detects the driver's face direction in real space based on the face information output from the face detection unit 12, specifically, based on the driver's face in the captured image detected by the face detection unit 12. The face direction detection unit 131 may detect the driver's face direction using, for example, a known face direction detection technique for detecting the face direction from the captured image. The driver's face direction is represented, for example, by at least one of a yaw angle, which is a rotation angle about an axis in the vertical direction of the vehicle, a pitch angle, which is a rotation angle about an axis in the horizontal direction of the vehicle, and a roll angle, which is a rotation angle about an axis in the longitudinal direction of the vehicle, which are predetermined references. The face direction detection unit 131 outputs information related to the detected driver's face direction (hereinafter referred to as "face direction information") to the reference posture estimation unit 14 and the posture imbalance determination unit 18. The face direction information includes at least one of information on the yaw angle, the pitch angle, and the roll angle.

[0027] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12, more specifically, based on the face of the driver in the captured image detected by the face detection unit 12. In the first embodiment, the driver's head position in the captured image is indicated, for example, by the center of the driver's eyebrows. The head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's eyebrows in the captured image as the driver's head position. Note that this is merely an example, and the driver's head position in the captured image may also be indicated, for example, by the center of the driver's facial area or the center of a line connecting the inner corners of the driver's eyes. In this case, the head position detection unit 132 detects, for example, a point in real space corresponding to the center of the driver's facial area in the captured image or the center of a line connecting the inner corners of the driver's eyes as the driver's head position in real space.

[0028] The head position detection unit 132 may detect the driver's head position using, for example, a known coordinate transformation technique that transforms points on a captured image into points in real space. The driver's head position is represented, for example, by coordinates in real space. The head position detection unit 132 outputs information related to the detected driver's head position (hereinafter referred to as "head position information") to the reference posture estimation unit 14 and the posture collapse determination unit 18. The head position information includes coordinate information of the driver's head position.

[0029] The detection reliability calculation unit 133 calculates a detection reliability indicating the reliability that the detected driver posture is at least one of a predetermined facial orientation and a predetermined head position. The detection reliability calculation unit 133 then outputs the calculated detection reliability to at least one of the reference posture estimation unit 14 and the posture deviation determination unit 18. The detection reliability can be calculated, for example, using the method disclosed in the document "Rapid Object Detection using a Boosted Cascade of Simple Features", 2001, Paul Viola, et al. In this document, the detection reliability of at least one of a facial orientation and a head position is calculated from the degree of match between at least one of a facial orientation and a head position learned in advance from a large number of facial images and at least one of a facial orientation and a head position extracted from an image of the driver.

[0030] In addition, even if the method is not the method described in the above document, the detection reliability calculation unit 133 can use any method that can detect a specific object corresponding to at least one of the face direction and head position from an image and calculate the detection reliability.

[0031] When the detected driver posture is less than a first threshold, the reference posture estimation unit 14 determines that the detected driver posture can be considered to be the reference posture of the driver (hereinafter referred to as detected driver posture A), and when the detected driver posture is equal to or greater than the first threshold, the reference posture estimation unit 14 determines that the detected driver posture can be considered to be a posture that the driver has changed from the reference posture (hereinafter referred to as detected driver posture B).

[0032] Then, the most frequent detected driver posture, which is the detected driver posture A and the detected driver posture B that is detected most frequently by the feature detection unit 13 during a predetermined period (hereinafter referred to as the "reference setting period"), is estimated as the driver's reference posture.

[0033] The reference setting period is, for example, the period from when the vehicle ignition is turned on until a predetermined time (e.g., 10 seconds) has elapsed. Note that this is merely an example, and the reference setting period may be, for example, a predetermined time (e.g., 3 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km / h) and the steering angle is within a predetermined range (e.g., ±20 degrees).

[0034] In addition, the detected driver posture is used to estimate at least one of the driver's reference facial orientation and head position as the reference posture, but the reference posture may also be estimated using the detected driver posture and at least one of the detection reliability of the facial orientation and the detection reliability of the head position.

[0035] The reference posture estimation unit 14 outputs a reference posture (hereinafter referred to as a first reference posture) estimated using the most frequent value in the captured image to the reference posture storage unit 15, and after outputting the first reference posture to the reference posture storage unit 15, the reference posture estimation unit 14 outputs a reference posture (hereinafter referred to as a second reference posture) estimated using the most frequent value in a captured image different from the captured image from which the first reference posture output to the reference posture storage unit 15 was estimated to the reference posture candidate storage unit 16. Furthermore, the reference posture estimation unit 14 acquires the first reference posture from the reference posture storage unit 15 as a reference posture to be output to the posture collapse determination unit 18.

[0036] The facial orientation in the first reference posture and the second reference posture is expressed by at least one of a yaw angle, a pitch angle, and a roll angle. The head position in the first reference posture and the second reference posture is expressed by, for example, coordinates in real space. The reference posture information includes coordinate information of the driver's head position as information on the driver's head position.

[0037] FIG. 2 shows a graph illustrating the relationship between the detected driver posture and the frequency with which each value of the detected driver posture is detected by the feature detection unit 13 during a predetermined period (hereinafter referred to as the detection frequency), which is used when the reference posture estimation unit 14 estimates the driver's reference posture.

[0038] The vertical axis indicates the detection frequency, which takes values ​​from 0 to 35 in FIG. 2 . The horizontal axis indicates the detected driver posture, which takes values ​​from 150 to 250 in FIG. 2 . Here, the detected driver posture in FIG. 2 refers to the driver's head position detected by the feature detection unit 13. Furthermore, when the driver is positioned in a specific head position, the detection frequency of the specific head position is higher than other head positions. Here, the detected driver posture is classified into detected driver posture A and detected driver posture B. For example, if the first threshold is 240, when the detected driver posture is less than the first threshold, it is classified as detected driver posture A, and when the detected driver posture is equal to or greater than the first threshold, it is classified as detected driver posture B. In FIG. 2 , the detected driver posture A is 150 to 230, and the detected driver posture B is 240 to 250.

[0039] The reference posture estimation unit 14 estimates the most frequent value of the detected driver posture as the first reference posture from among the detected driver postures A and B shown in Fig. 2. In Fig. 2, the reference posture estimation unit 14 estimates 190 as the first reference posture and outputs the estimated first reference posture to the reference posture storage unit 15.

[0040] The reference attitude storage unit 15 stores the first reference attitude output from the reference attitude estimation unit 14. When the reference attitude update determination unit 17, which will be described later, determines to update the first reference attitude to the second reference attitude, the first reference attitude is updated to the second reference attitude.

[0041] The reference attitude candidate storage unit 16 stores the second reference attitude output from the reference attitude estimation unit 14 .

[0042] The reference attitude update determination unit 17 updates the first reference attitude to the second reference attitude when the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 is equal to or greater than a second threshold, and does not update the first reference attitude when the difference is less than the second threshold.

[0043] FIG. 3 shows a graph illustrating the relationship between the detected driver posture when the driver changes posture from the reference posture after the reference posture estimation unit 14 estimates the reference posture in the graph shown in FIG. 2 and the detection frequency of each value of the detected driver posture over a predetermined period. The vertical axis indicates the detection frequency, which ranges from 0 to 35 in FIG. 3. The horizontal axis indicates the detected driver posture, which ranges from 150 to 250 in FIG. 3. Here, the detected driver posture in FIG. 3 refers to the driver's head position detected by the feature detection unit 13. Furthermore, when the driver is positioned in a specific head position, the detection frequency of that specific head position is higher than for other head positions. Furthermore, similar to FIG. 2, the detected driver posture is classified as detected driver posture A or detected driver posture B.

[0044] The reference posture update determination by the reference posture update determination unit 17 in the situation shown in Fig. 3 will be described. In Fig. 3, the reference posture estimation unit 14 estimates the most frequent value of the detected driver postures, which are the detected driver postures A and B shown in Fig. 3, as the first reference posture. In Fig. 3, the reference posture estimation unit 14 outputs 240 to the reference posture candidate storage unit 16 as the second reference posture.

[0045] Here, the reference attitude update determination unit 17 determines whether to update the first reference attitude by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with a second threshold value.

[0046] For example, if the second threshold is 45, and the difference is equal to or greater than the second threshold, the first reference attitude is updated to the second reference attitude. If the difference is less than the second threshold, the first reference attitude is not updated. Here, if the first reference attitude is 190 and the second reference attitude is 240, the difference is 50, which satisfies the condition of being equal to or greater than the second threshold. Therefore, the first reference attitude is updated to the second reference attitude 240.

[0047] FIG. 4 shows a graph illustrating the relationship between the detected driver posture in situation A and the detection frequency of each value of the detected driver posture over a predetermined period after the reference posture estimation unit 14 estimates the reference posture in the graph shown in FIG. 2 . The vertical axis indicates the detection frequency, which ranges from 0 to 35 in FIG. 4 . The horizontal axis indicates the detected driver posture, which ranges from 150 to 250 in FIG. 4 . Here, the detected driver posture in FIG. 4 refers to the driver's head position detected by the feature detection unit 13. Furthermore, when the driver is positioned in a specific head position, the detection frequency of the specific head position is higher than that of other head positions. Furthermore, as in FIG. 2 , the detected driver posture is classified as detected driver posture A or detected driver posture B. Here, situation A refers to a situation in which the driver's posture is actually positioned in the reference posture and the driver does not change posture from the reference posture, but the detected driver posture fluctuates between detected driver posture A and detected driver posture B due to vehicle vibration, etc.

[0048] In the case of situation A, both the detection frequency of detected driver posture A (detection frequency of 230 in FIG. 4 ) and the detection frequency of detected driver posture B (detection frequency of 240 in FIG. 4 ) may be equal to or higher than a predetermined frequency. Also, in the case of situation A, the detection frequency of detected driver posture A (detection frequency of 230 in FIG. 4 ) may be higher than the detection frequency of detected driver posture B (detection frequency of 240 in FIG. 4 ). In other words, even though the detection frequency of detected driver posture A is higher than the detection frequency of detected driver posture B, the detection frequency of detected driver posture B may be equal to or higher than a predetermined frequency.

[0049] The reference posture update determination unit 17 determines whether to update the reference posture in the situation shown in Fig. 4. The reference posture estimation unit 14 estimates the most frequent detected driver posture as the second reference posture from among the detected driver postures consisting of detected driver posture A and detected driver posture B shown in Fig. 4. In Fig. 4, the reference posture estimation unit 14 estimates 230 as the second reference posture and outputs the estimated second reference posture to the reference posture candidate storage unit 16.

[0050] Here, the reference attitude update determination unit 17 determines whether to update the first reference attitude by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with a second threshold value.

[0051] For example, if the second threshold is 45, and the difference is equal to or greater than the second threshold, the first reference attitude is updated to the second reference attitude. If the difference is less than the second threshold, the first reference attitude is not updated. Here, if the first reference attitude is 190 and the second reference attitude is 230, the difference is 40, which does not satisfy the condition of being equal to or greater than the second threshold. Therefore, the first reference attitude of 190 is not updated.

[0052] The posture deviation determination unit 18 determines whether the driver's posture corresponds to any one of a plurality of predetermined posture deviation types (hereinafter referred to as "posture deviation types") by comparing the detected driver posture with the reference posture acquired from the reference posture estimation unit 14. In the first embodiment, the determination by the posture deviation determination unit 18 of whether the driver's posture corresponds to any one of the plurality of posture deviation types is also referred to as "posture deviation type determination." The plurality of posture deviation types are set in advance as types of postures that are considered to be abnormal postures caused by the driver being in an abnormal state.

[0053] FIG. 5 (partially modified from the "Basic Design Document for Driver Abnormality Automatic Detection System" by the Advanced Safety Vehicle Promotion Study Group, Road Transport Bureau, Ministry of Land, Infrastructure, Transport and Tourism, March 2018) is a diagram for explaining multiple types of poor posture. As shown in FIG. 5, there are multiple types of poor posture. FIG. 5 illustrates the following: "head down" in which the driver leans forward and maintains a posture in which their face is close to the steering wheel; "head down" in which the driver maintains a posture in which their face is facing downward; "backward lean" in which the driver's upper body leans backward and maintains a posture in which their face is facing upward; "backward lean" in which the driver's upper body leans upward and maintains a posture in which their face is facing upward; "head tilted to the side only" in which the driver's face is tilted to the left or right; "sideways lean" in which the driver's upper body leans to the left or right and maintains a posture in which their face is tilted in the same direction; and "leaning to the side" in which the driver's upper body leans to the left or right. In the present disclosure, the multiple pre-set posture types are "leaning face down," "head down," "arching backward," "arching back," "head tilted to the side," "head tilted to the side," and "leaning to the side," as shown in Fig. 5. Information on the multiple posture types is stored in a location that can be referenced by the posture determination unit 18.

[0054] Specifically, the posture deviation determination unit 18 determines which of a plurality of posture deviation types the driver belongs to depending on how much the detected driver posture has changed compared to a first reference posture. Note that the conditions for determining which posture deviation type the driver belongs to based on how much the detected driver posture has changed compared to the first reference posture are set in advance and stored in the posture deviation determination unit 18. The posture deviation determination unit 18 may perform posture deviation type determination using a learning device such as an SVM (Support Vector Machine), for example.

[0055] The posture error determination unit 18 outputs the posture error type determination result to the abnormal posture determination unit 19. The posture error type determination result includes information on whether or not the driver's posture corresponds to any of the posture error types.

[0056] The abnormal posture determination unit 19 uses the posture error type determination result output by the posture error determination unit 18 to determine whether the driver's posture is abnormal or not.

[0057] If the posture error determination unit 18 determines that the driver's posture does not fall into any of the posture error types, the abnormal posture determination unit 19 determines that the driver is not in an abnormal posture.

[0058] When the posture error determination unit 18 determines that the driver's posture is an abnormal posture, the abnormal posture determination unit 19 determines the duration of the state in which the driver's posture is an abnormal posture. For example, the abnormal posture determination unit 19 associates the acquired range determination result or posture error type determination result with information on the date and time when the range determination result or posture error type determination result was acquired, and stores them in a storage unit (not shown). The abnormal posture determination unit 19 can determine the duration of the state in which the driver's posture is an abnormal posture from the stored range determination result or posture error type determination result. When the duration of the state in which the driver's posture is an abnormal posture reaches a preset threshold (hereinafter referred to as the "abnormal posture determination threshold"), the abnormal posture determination unit 19 determines that the driver has an abnormal posture.

[0059] The abnormal posture determination unit 19 outputs the determination result as to whether or not the driver is in an abnormal posture (hereinafter referred to as the “abnormal posture determination result”) to the output control unit 20 .

[0060] When the abnormal posture determination unit 19 determines that the driver is in an abnormal posture, the output control unit 20 outputs warning information to the output device 3 to warn the driver.

[0061] The operation of the occupant state determination device 1 according to the first embodiment will now be described. Fig. 6 is a flowchart for explaining the operation of the occupant state determination device 1 according to the first embodiment. The operation of the occupant state determination device 1 is started, for example, after the ignition of the vehicle is turned on. It is also preferable to perform the occupant state determination process every time an image is acquired from the imaging device 2, that is, for each frame.

[0062] The image acquisition unit 11 acquires a captured image from the imaging device 2 (step ST1). The image acquisition unit 11 outputs the acquired captured image to the face detection unit 12.

[0063] The face detection unit 12 detects the driver's face in the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). The face detection unit 12 outputs face information to the feature detection unit 13.

[0064] The face direction detection unit 131 detects the face direction of the driver in real space based on the face information output from the face detection unit 12 in step ST2, specifically based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The face direction detection unit 131 outputs the face direction information to the reference posture estimation unit 14 and the posture collapse determination unit 18.

[0065] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, more specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and the posture collapse determination unit 18.

[0066] The head position detection unit 132 detects the head position of the driver in real space based on the face information output from the face detection unit 12 in step ST2, more specifically, based on the driver's face in the captured image detected by the face detection unit 12 (step ST3). The head position detection unit 132 outputs the head position information to the reference posture estimation unit 14 and the posture collapse determination unit 18.

[0067] The detection reliability calculation unit 133 calculates the detection reliability of at least one of the face orientation and the head position based on the face information output from the face detection unit 12 in step ST2, specifically based on the driver's face in the captured image detected by the face detection unit 12 (step ST4). The detection reliability calculation unit 133 outputs the detection reliability to at least one of the reference posture estimation unit 14 and the posture collapse determination unit 18.

[0068] If the detected driver posture is less than the first threshold, the reference posture estimation unit 14 determines it to be detected driver posture A, and if the detected driver posture is greater than or equal to the first threshold, it determines it to be detected driver posture B (step ST5).

[0069] The reference posture estimation unit 14 estimates the most frequent detected driver posture, which is the detected driver posture A and detected driver posture B that is detected most frequently by the feature detection unit 13, as the driver's reference posture (step ST6).

[0070] If the reference posture estimation unit 14 has completed estimating the driver's reference posture in the previous frame, in other words, after outputting the first reference posture to the reference posture storage unit 15 (if "YES" in step ST7), the reference posture estimation unit 14 outputs a second reference posture estimated using the most frequent value in an image captured different from the image captured using the first reference posture estimated and output to the reference posture candidate storage unit 16 (step ST8).

[0071] If the reference posture estimation unit 14 has not completed estimating the driver's reference posture in the previous frame (if "NO" in step ST7), the reference posture estimation unit 14 outputs the first reference posture in the reference posture estimated using the most frequent value in the captured image to the reference posture memory unit 15 (step ST9).

[0072] When the reference posture estimation unit 14 estimates the driver's reference posture, it sets a reference posture estimation completion flag to "1." The initial value of the reference posture estimation completion flag is "0," and the reference posture estimation completion flag is initialized, for example, when the ignition of the occupant state determination device 1 is turned off. The reference posture estimation completion flag is stored in a location that can be referenced by the occupant state determination device 1 (step ST10).

[0073] If the difference between the first reference posture stored in the reference posture memory unit 15 and the second reference posture stored in the reference posture candidate memory unit 16 is greater than or equal to a second threshold value (if "YES" in step ST11), the reference posture update determination unit 17 updates the first reference posture to the second reference posture (step ST12).

[0074] If the difference between the first reference attitude and the second reference attitude is less than the second threshold value ("NO" in step ST11), the reference attitude update determination unit 17 does not update the first reference attitude (step ST13).

[0075] The posture error determination unit 18 determines whether the driver's posture corresponds to any of a plurality of predetermined posture error types by comparing the detected driver posture with the reference posture acquired from the reference posture estimation unit 14 (step ST14). The posture error determination unit 18 outputs the posture error type determination result to the abnormal posture determination unit 19.

[0076] If the posture imbalance determination unit 18 determines that the driver's posture does not fall into any of the posture imbalance types (if "NO" in step ST14), the abnormal posture determination unit 19 determines that the driver is not in an abnormal posture (step ST15).

[0077] If the posture imbalance determination unit 18 determines that the driver's posture is imbalance (if "YES" in step ST14), the abnormal posture determination unit 19 determines the duration of the driver's posture imbalance (step ST16).

[0078] If the duration of the state in which the driver's posture is poor reaches the threshold for determining abnormal posture (if "YES" in step ST16), the abnormal posture determination unit 19 determines that the driver has an abnormal posture and causes the output device 3 to output a warning that the driver's posture is abnormal (step ST17). If the duration of the state in which the driver's posture is poor does not reach the threshold for determining abnormal posture (if "NO" in step ST16), the abnormal posture determination unit 19 determines that the driver does not have an abnormal posture.

[0079] 7 and 8 are diagrams illustrating an example of the hardware configuration of the occupant state determination device 1 according to the first embodiment. In the first embodiment, the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 are realized by a processing circuit 1001. That is, the occupant state determination device 1 includes the processing circuit 1001 for performing control to determine whether the posture of a vehicle occupant is an abnormal posture based on a captured image. The processing circuit 1001 may be dedicated hardware as shown in FIG. 7, or a processor 1004 that executes a program stored in a memory 1005 as shown in FIG. 8. The memory 1000 includes the reference posture estimation unit 14 and the reference posture storage unit 15.

[0080] When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0081] When the processing circuit is the processor 1004, the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1005. The processor 1004 reads and executes the program stored in the memory 1005 to execute the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20. In other words, the occupant state determination device 1 includes the memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST16 of FIG. 7 described above. In addition, the program stored in memory 1005 can also be said to cause the computer to execute the processing procedures or methods of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20. Here, memory 1005 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or a magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0082] It is also possible to realize some of the functions of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 with dedicated hardware and some with software or firmware. For example, the functions of the image acquisition unit 11 and output control unit 20 can be realized by a processing circuit 1001 as dedicated hardware, and the functions of the face detection unit 12, feature detection unit 13, reference posture estimation unit 14, posture collapse determination unit 18, and abnormal posture determination unit 19 can be realized by the processor 1004 reading and executing programs stored in memory 1005.

[0083] The storage unit (not shown) is, for example, a memory 1005. The occupant state determination device 1 also includes an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication with devices such as the imaging device 2 or the output device 3.

[0084] As described above, according to the first embodiment, the occupant state determination device 1 includes a face detection unit 12 that detects the face of a vehicle occupant in a captured image in which the face of the occupant is captured; a feature detection unit 13 that detects a detected occupant posture, which is at least one of the occupant's facial orientation and head position, using the occupant's face in the captured image detected by the face detection unit 12; and a mode that is a detected occupant posture that can be considered to be a reference posture, which is at least one of the facial orientation and head position serving as a reference for the occupant, when the detected occupant posture detected by the feature detection unit 13 is less than a first threshold value, and that can be considered to be a detected occupant posture that can be considered to be a posture that the occupant has changed from the reference posture when the detected occupant posture is equal to or greater than the first threshold value, and a mode that is a detected occupant posture that is most frequently detected by the feature detection unit 13 among the detected occupant postures that can be considered to be the reference posture and the detected occupant postures that can be considered to be a posture that the occupant has changed from the reference posture, which are acquired during a predetermined period. a reference attitude update determination unit 17 that updates the first reference attitude to the second reference attitude if the difference between a first reference attitude, which is a reference attitude estimated using the most frequent value in the captured image, and a second reference attitude, which is a reference attitude estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and does not update the first reference attitude if the difference is less than the second threshold; a posture collapse determination unit 18 that performs posture collapse type determination to determine which of a plurality of posture collapse types the occupant's posture falls into by comparing the detected occupant posture with the first reference attitude; and an abnormal posture determination unit 19 that determines that the occupant has an abnormal posture if the posture collapse determination unit 18 determines that the occupant's posture falls into any of the plurality of posture collapse types, and determines that the occupant does not have an abnormal posture if the posture collapse determination unit 18 determines that the occupant's posture does not fall into any of the plurality of posture collapse types.

[0085] The reference posture estimation unit 14 estimates the most frequent detected driver posture, which is the detected driver posture A and the detected driver posture B that are detected most frequently by the feature detection unit 13, as the driver's reference posture during the reference setting period.

[0086] In addition, the reference attitude update determination unit 17 determines whether or not to update the first reference attitude stored in the reference attitude memory unit 15 by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with a second threshold value.

[0087] With this configuration, the occupant state determination device 1 updates the reference posture when the driver changes his / her posture intentionally or due to habit, for example, during a long drive or due to changes in the driving environment (changes in weather or time of day). Therefore, the amount of change from the reference posture used to determine whether or not the posture is poor can be calculated with high accuracy. As a result, it is possible to prevent the occurrence of poor posture detection, which is determined not to be poor posture due to the inability to accurately calculate the amount of change, even though the posture is actually poor, and improve the accuracy of occupant state detection.

[0088] Furthermore, in the situation A shown in Figure 4, if an attempt is made to update the reference posture using Patent Document 1 as described above, even if the occupant is in the reference posture, it may be determined that the occupant has changed their posture from the reference posture, and the occupant's reference posture may be updated. This makes it difficult to accurately calculate the amount of change from the reference posture used to determine whether or not posture has been lost. As a result, even if posture has been lost, the amount of change cannot be accurately calculated, resulting in an undetected posture that is determined not to be posture loss, which reduces the accuracy of detecting the driver's inability to drive.

[0089] On the other hand, the occupant state determination device 1 estimates as the driver's reference posture the most frequent detected driver posture, which is the detected driver posture A and detected driver posture B that is detected most frequently by the feature detection unit 13 during the reference setting period, by the reference posture estimation unit 14.

[0090] By adopting such a configuration, the detected driver posture corresponding to the detected driver posture A in situation A is excluded from the target of the most frequent value calculation in the reference posture estimation unit 14, and it is possible to avoid only the detected driver posture including the detected driver posture B being the target of the most frequent value calculation in the reference posture estimation unit 14.

[0091] As a result, it is possible to prevent differences from occurring in at least one of the facial direction and head position between the first reference posture updated by the reference posture update determination unit 17 and the actual driver posture, and it is possible to avoid the driver's reference posture being updated because it is determined that the driver has changed posture from the reference posture even when the driver is in the reference posture.

[0092] Therefore, the amount of change from the reference posture used to determine whether or not the posture is out of alignment can be calculated with high accuracy. As a result, even though the posture is actually out of alignment, the amount of change cannot be calculated with high accuracy, so that the occurrence of undetected posture is prevented, and the accuracy of detecting the state of the occupant can be improved.

[0093] In addition, in the above-described first embodiment, the reference attitude update determination unit 17 determines whether to update the first reference attitude by comparing the difference between the first reference attitude stored in the reference attitude memory unit 15 and the second reference attitude stored in the reference attitude candidate memory unit 16 with the second threshold value. However, the reference attitude update determination unit 17 may determine whether to update the first reference attitude by machine learning.

[0094] Furthermore, in the above-described embodiment 1, the most frequent detected driver posture, which is the detected driver posture A and the detected driver posture B that is detected most frequently by the feature detection unit 13, is estimated as the driver's reference posture. However, the detected driver posture A and the detected driver posture B that is detected by the feature detection unit 13 with a frequency equal to or greater than a predetermined threshold may also be estimated as the reference posture.

[0095] In the first embodiment, the vehicle occupant whose posture is to be determined by the occupant state determination device 1 is the driver of the vehicle, but this is merely an example. The occupant state determination device 1 can determine whether or not any other vehicle occupant other than the driver has an abnormal posture.

[0096] In the first embodiment described above, the occupant state determination device 1 is an in-vehicle device mounted on a vehicle, and the in-vehicle device is provided with the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture storage unit 15, reference posture candidate storage unit 16, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20. However, without being limited to this, some of the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture storage unit 15, reference posture candidate storage unit 16, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 may be mounted on the in-vehicle device of the vehicle, and the others may be provided in a server connected to the in-vehicle device via a network, so that a system is formed by the in-vehicle device and the server. In addition, the image acquisition unit 11, face detection unit 12, feature detection unit 13, reference posture estimation unit 14, reference posture storage unit 15, reference posture candidate storage unit 16, reference posture update determination unit 17, posture collapse determination unit 18, abnormal posture determination unit 19, and output control unit 20 may all be provided on the server.

[0097] 1 Occupant state determination device, 2 Imaging device, 3 Output device, 11 Image acquisition unit, 12 Face detection unit, 13 Feature detection unit, 131 Face direction detection unit, 132 Head position detection unit, 133 Detection reliability calculation unit, 14 Reference posture estimation unit, 15 Reference posture storage unit, 16 Reference posture candidate storage unit, 17 Reference posture update determination unit, 18 Posture collapse determination unit, 19 Abnormal posture determination unit, 20 Output control unit, 100 Occupant state determination system, 1001 Processing circuit, 1002 Input interface device, 1003 Output interface device, 1004 Processor, 1005 Memory

Claims

1. a face detection unit that detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit that detects a detected occupant posture, which is at least one of a facial direction and a head position of the occupant, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit that, when the detected occupant posture detected by the feature detection unit is less than a first threshold, determines the detected occupant posture as a reference posture which is at least one of a facial orientation and a head position serving as a reference for the occupant, and, when the detected occupant posture is equal to or greater than the first threshold, determines the detected occupant posture as a posture which the occupant can be considered to have changed from the reference posture, and estimates, as the reference posture, a mode which is the detected occupant posture detected most frequently by the feature detection unit among the detected occupant postures which the occupant can be considered to be the reference posture and the detected occupant postures which the occupant can be considered to have changed from the reference posture, which are acquired during a predetermined period; a reference orientation update determination unit that updates the first reference orientation to the second reference orientation when a difference between a first reference orientation, which is the reference orientation estimated using the most frequent value in the captured image, and a second reference orientation, which is the reference orientation estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and does not update the first reference orientation when the difference is less than the second threshold; a posture imbalance determination unit that performs a posture imbalance type determination as to which of a plurality of posture imbalance types the posture of the occupant corresponds by comparing the detected occupant posture with the first reference posture; an abnormal posture determination unit that determines that the occupant is in an abnormal posture when the posture determination unit determines that the posture of the occupant corresponds to any one of a plurality of posture deviation types, and determines that the occupant is not in an abnormal posture when the posture determination unit determines that the posture of the occupant does not correspond to any one of a plurality of posture deviation types; An occupant state determination device comprising:

2. the facial orientation of the occupant is represented by at least one of a yaw angle, which is a rotation angle about an axis in a vertical direction of the vehicle, a pitch angle, which is a rotation angle about an axis in a horizontal direction of the vehicle, and a roll angle, which is a rotation angle about an axis in a front-rear direction of the vehicle, which are predetermined references; The head position of the occupant is obtained by converting a point arbitrarily set on the face of the occupant in the captured image detected by the face detection unit into a point in real space. The occupant condition determination device according to claim 1 .

3. the reference attitude update determination unit determines, by machine learning, whether to update the first reference attitude to the second reference attitude or not to update the first reference attitude. The occupant condition determination device according to claim 1 or 2.

4. 3. The occupant state determination device according to claim 1, further comprising an output control unit that outputs warning information for warning the occupant when the abnormal posture determination unit determines that the occupant is in the abnormal posture.

5. The occupant state determination device according to claim 1 or 2, An abnormal posture determination system including: an imaging device installed in the vehicle so as to be able to image at least a face of the occupant; and an output device mounted in the vehicle and outputting a warning that the posture of the occupant is the abnormal posture, The occupant state determination device determines whether or not the posture of the occupant is the abnormal posture by using the captured image captured by the imaging device, and when it is determined that the posture of the occupant is the abnormal posture, causes the output device to output a warning that the posture of the occupant is the abnormal posture. Occupant status determination system.

6. A face detection unit detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit detecting a detected occupant posture, which is at least one of a facial direction and a head position of the occupant, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit estimating, when the detected occupant posture detected by the feature detection unit is less than a first threshold, a detected occupant posture that can be regarded as a reference posture which is at least one of a facial orientation and a head position serving as a reference for the occupant, and when the detected occupant posture is equal to or greater than the first threshold, a detected occupant posture that can be regarded as a posture in which the occupant has changed from the reference posture, and estimating, as the reference posture, a mode that is the detected occupant posture that has been detected most frequently by the feature detection unit among the detected occupant postures that can be regarded as the reference posture of the occupant and the detected occupant postures that can be regarded as a posture in which the occupant has changed from the reference posture, which are acquired during a predetermined period; a reference orientation update determination unit updating the first reference orientation to the second reference orientation when a difference between a first reference orientation, which is the reference orientation estimated using the most frequent value in the captured image, and a second reference orientation, which is the reference orientation estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and not updating the first reference orientation when the difference is less than the second threshold; a posture deviation determination unit performing a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant corresponds to by comparing the detected occupant posture with the first reference posture; and an abnormal posture determination unit determining that the occupant has an abnormal posture when the posture deviation determination unit determines that the posture of the occupant corresponds to any of a plurality of posture deviation types, and determining that the occupant does not have an abnormal posture when the posture deviation determination unit determines that the posture of the occupant does not correspond to any of a plurality of posture deviation types. An occupant state determination method comprising:

7. A face detection unit detects a face of a vehicle occupant in a captured image in which the face of the vehicle occupant is captured; a feature detection unit detecting a detected occupant posture, which is at least one of a facial direction and a head position of the occupant, by using the face of the occupant in the captured image detected by the face detection unit; a reference posture estimation unit estimating, when the detected occupant posture detected by the feature detection unit is less than a first threshold, a detected occupant posture that can be regarded as a reference posture which is at least one of a facial orientation and a head position serving as a reference for the occupant, and when the detected occupant posture is equal to or greater than the first threshold, a detected occupant posture that can be regarded as a posture in which the occupant has changed from the reference posture, and estimating, as the reference posture, a mode that is the detected occupant posture that has been detected most frequently by the feature detection unit among the detected occupant postures that can be regarded as the reference posture of the occupant and the detected occupant postures that can be regarded as a posture in which the occupant has changed from the reference posture, which are acquired during a predetermined period; a reference orientation update determination unit updating the first reference orientation to the second reference orientation when a difference between a first reference orientation, which is the reference orientation estimated using the most frequent value in the captured image, and a second reference orientation, which is the reference orientation estimated using the most frequent value in a captured image different from the captured image, is equal to or greater than a second threshold, and not updating the first reference orientation when the difference is less than the second threshold; a posture deviation determination unit performing a posture deviation type determination as to which of a plurality of posture deviation types the posture of the occupant corresponds by comparing the detected occupant posture with the first reference posture; An abnormal posture determination unit determines that the occupant has an abnormal posture if the posture collapse determination unit determines that the posture of the occupant falls into any of a plurality of posture collapse types, and determines that the occupant does not have an abnormal posture if the posture collapse determination unit determines that the posture of the occupant does not fall into any of a plurality of posture collapse types.